Load balanced access to distributed scaling endpoints using global network addresses

ABSTRACT

Systems and methods are described to enable the load-balanced use of globalized network addresses, addressable throughout a network to access a network-accessible service. A set of global access points are provided, which advertise availability of the globalized network addresses. On receiving a request to access a network-accessible service, a global access point can select an endpoint for the service from among a number of data centers, based on a desired distribution of traffic among the data centers. The access point is responsive to scaling that might occur at endpoints by periodically retrieving updated configuration information for the endpoints, enabling the access point to nimbly respond to changes in endpoints for the service.

BACKGROUND

Generally described, computing devices utilize a communication network,or a series of communication networks, to exchange data. Companies andorganizations operate computer networks that interconnect a number ofcomputing devices to support operations or provide services to thirdparties. The computing systems can be located in a single geographiclocation or located in multiple, distinct geographic locations (e.g.,interconnected via private or public communication networks).Specifically, data centers or data processing centers, herein generallyreferred to as “data centers,” may include a number of interconnectedcomputing systems to provide computing resources to users of the datacenter. The data centers may be private data centers operated on behalfof an organization or public data centers operated on behalf, or for thebenefit of, the general public.

Because the resources of any individual computing device are limited, avariety of techniques are known to attempt to balance resource usageamong devices. For example, a “load balancer” device may be placedbetween client devices requesting use of computing resources and serversproviding such resources. The load balancer may attempt to distributethe requests among the servers, thus allowing the servers to work inconjunction to provide such resources. A significant drawback of use ofsuch a load balancer is that it may create a single point of failure,such that if the load balancer fails, requests to the servers fail, andresources are made unavailable.

Another type of load balancing known in the art is domain name system(DNS) load balancing. DNS generally refers to a network of devices thatoperate to translate human-recognizable domain names into networkaddress on a network, such as the Internet. To load balance using DNS, aDNS server is populated with network addresses of multiple computingdevices offering a given resource. When responding to requests toresolve a domain name associated with that resource, the DNS serveralternates which addresses are provided, thus causing requests for theresources to be divided among the provided addresses. Because the DNSserver does not act as a conduit for traffic to the resources, andbecause DNS generally provides highly redundant operation, DNS loadbalancing generally avoids having a single point of failure. However, asignificant drawback of DNS load balancing is the delay required toalter a load balancing scheme. DNS requests generally flow through aseries of DNS resolvers, each of which may cache prior DNS results for aperiod. Thus, changes made at a DNS server in an attempt to alter howload is balanced among servers can take significant time to propagate.Particularly in instances where a server has failed, these delays cancause significant errors in network communications. Moreover,maintaining up-to-date DNS records can be difficult, as they generallymust be modified as new devices are added to or removed from a loadbalancing scheme.

Yet another type of load balancing known in the art is the use of“anycast” network addresses. In a network, such as the Internet,different autonomous systems (“AS”) provide devices with differentnetwork addresses. Each AS notifies it's neighboring AS's of addressesavailable within its network, by “advertising” the addresses. Mosttypically, each address is associated with a single location (e.g., asingle device or group of devices). In an anycast configuration,multiple devices, often in multiple AS's, advertise the same networkaddress. Depending on the configuration of neighboring AS's, clientrequests to access the address can then be routed to any of the multipledevice, distributing load among the devices. A significant drawback ofthe use of anycast to attempt to load balance is that routing to theanycasted address is generally controlled by neighboring networks. Thesenetworks are often under control of other entities. Thus, it isdifficult or impossible to completely control how requests aredistributed among devices with an anycasted network address. Moreover,when the configuration of neighboring networks changes, the distributionof requests may also change, leading to volatility in the load balancingscheme.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting an illustrative logical networkincluding multiple client devices and data centers, as well as a set ofglobal access points providing load-balanced access to the data centersfrom a set of global network addresses.

FIG. 2 is a block diagram depicting an illustrative configuration of adata center of FIG. 1.

FIG. 3 is a block diagram depicting an illustrative configuration of aglobal access point of FIG. 1.

FIG. 4 is a block diagram depicting an illustrative configuration of aflow manager server implementing one or more flow managers within aglobal access point of FIG. 1.

FIG. 5 depicts illustrative interactions for routing a request from aclient device addressed to a global network address to a global accesspoint of FIG. 1.

FIG. 6 depicts illustrative interactions for routing a request from aglobal access point to a data center of FIG. 1, based at least partly onload-balancing requests among the data centers.

FIG. 7 depicts illustrative interactions for propagating endpointinformation from a data center to global access points of FIG. 1, suchthat the access points may correctly route traffic from client devicesaddressed to global network addresses.

FIG. 8 depicts an illustrative routine for increasing resiliency ofglobal network addresses, by selecting different neighboring devices towhich to advertise different global network addresses from access pointsof FIG. 1.

FIG. 9 depicts an illustrative routine for routing traffic addressed toa global network address associated with a service provided by endpointswithin data centers, by use of the access points of FIG. 1.

FIG. 10 depicts an illustrative routine for updating information at theglobal access points of FIG. 1 regarding endpoints of a data center thatprovide a network-accessible service.

FIG. 11 depicts illustrative interactions for establishing aTransmission Control Protocol (TCP) session at a global access point ofFIG. 1, and handing off the TCP session to an endpoint to enablecommunication of a client device and the endpoint via the TCP session.

FIG. 12 depicts an illustrative routine for establishing a TransmissionControl Protocol (TCP) session at a global access point of FIG. 1, andhanding off the TCP session to an endpoint to enable communication of aclient device and the endpoint via the TCP session.

DETAILED DESCRIPTION

Generally described, aspects of the present disclosure relate toproviding load-balanced access to a pool of computing devices spreadacross multiple geographic locations, using one or more global networkaddress. More specifically, aspects of the present disclosure relate toproviding a distributed set of access points reachable via globalnetwork addresses, which access points select and route requests toendpoint devices within the pool based at least partly on load balancingthe requests. In one embodiment, the access points utilize anycastrouting techniques to advertise availability of global network addressesassociated with the pool of computing devices, thereby attractingtraffic addressed to those addresses. On receiving a request to accessthe pool, an access point may select an appropriate endpoint within thepool based on factors such as network performance criteria to theendpoint and load on the endpoint. The access point may then act as aproxy, routing the request to the endpoint and facilitating furthercommunications between the endpoint and a requesting device. The accesspoint may implement a variety of techniques, as disclosed herein, toprovide resilient, efficient access to the pool of endpoint devices. Asdisclosed herein, the access points may be distributed among a widegeographic area, thus eliminating single points of failure within thesystem. Moreover, by utilizing anycast advertisements, the access pointsmay distribute requests among the pool even when the requests areaddressed to a single network address, avoiding complications and delaysassociated with other techniques, such as DNS-based load balancing. Byacting as a proxy between clients and the pool of devices, rather thanserving resources directly, the access points may control distributionof requests to the pool independently of how external devices choose toroute the requests to the anycasted address, thus avoiding detrimentsassociated with traditional anycast networking. Thus, embodimentsdisclosed herein significantly improve on prior load balancingtechniques.

Embodiments of the present disclosure may illustratively be implementedin a wide geographic area. In one embodiment, the present disclosure isimplemented on the worldwide Internet, and provides global internetprotocol (IP) addresses, such as IP version 4 (IPv4) or IP version 6(IPv6) addresses. Different data centers may exist in differentgeographic locations, and each data center may include one or moreendpoint devices providing access to a network-based service. Examplesof such services include, but are not limited to, web hosting, datastorage, on-demand compute services, and the like. The resources at eachdata center may be limited, and thus an operator of the network-basedservice may wish to distribute load among such services. To simplifyoperation of the network-based service (e.g., avoiding the complicationsrelated to DNS-based load balancing), it may be desirable to provide theoperator with a single set of relatively static network addresses forthe service, which network addresses are manageable independently of theindividual endpoints providing access to the service. Such networkaddresses are generally referred to herein as “global” networkaddresses. As used herein, the term “global” is intended to refer to thescope of the network address with relation to the service (e.g., thatthe network address applies to the entire service, rather thanindividual devices), and does not necessarily imply that such a networkaddress is accessible worldwide. Nevertheless, embodiments of thepresent disclosure may be implemented to provide global networkaddresses that are generally accessible from a worldwide network, suchas the Internet.

To provide global network addresses for a service, a system is disclosedthat provides a set of geographically distributed global access points.Similarly to as noted above, the term “global” access point in thiscontext is generally intended to refer to the access point providingaccess to the service generally, as opposed to individual endpoints, anddoes not necessarily imply that such access points exist worldwide(though such a configuration is possible). Generally, however, theaccess points are geographically distributed. In one embodiment, theglobal access points are located in geographic locations different andmore numerous than those of the data centers providing endpoints for theservice, decreasing the average network distance between the globalaccess points and client devices attempting to access the service.

Each global access point may utilize anycast techniques to advertise theavailability of the service via the one or more global network addressesassociated with the service. Illustratively, each access point mayutilize Border Gateway Protocol (“BGP”) to advertise the global networkaddresses, by include a BGP “speaker” to announce availability of theglobal network address to neighboring networks. The global access pointmay thus attract traffic address to the global network addresses. Asdisclosed in more detail below, the global access points may in someinstances “shape” BGP announcements to increase resiliency of anetwork-based service to network interruptions. For example, the globalaccess points may divide their announced global network addresses intotwo groups, and assign a service at least one network address from eachgroup. The global access points may then announce each group ofaddresses to a different neighboring network. In this manner, eachaccess point can effectively create two paths to reach the point:through a first neighbor using an address of the first group, or througha second neighbor using an address of a second group. Thus, if oneneighboring network fails in some way, an alternative path to reach theaccess point exists. Moreover, because each access point may operate inthis manner, if one access point fails entirely, traffic may beautomatically routed to another access point via traditional anycastrouting mechanisms. While examples are provided herein related to twogroups, any number of groups may be provided.

To provide even further resiliency, each access point may in someinstances segregate traffic according to a specific subset of theadvertised global network addresses, and utilize different devices orprogram threads to process traffic addressed to each subset. In thisway, if traffic of one subset is problematic to operation of the accesspoint (e.g., due to misconfiguration related to a global network addressof the subset), operation of a device or thread handling traffic ofanother subset is unlikely to be effected. In one embodiment, the accesspoint uses a combination of different network address groups, andsegregated processing of subsets of each network group. Moreover,services may be “shuffled” among the subsets of each group, such that iftwo services are assigned network addresses within a common subset ofone group, the two services are likely to be assigned network addresseswithin different subsets of a second group. Under this configuration, ifa specific service's configuration causes issues to operation of anaccess point with respect to a subset of one group, other services wouldlikely still be reachable via an alternative group (where theiraddresses are unlikely to be on the same subset as the problematicservice). By increasing the number of address groups and subsets, thetotal number of services effected by a problematic service can beprogressively decreased.

After receiving a request to access a service, a global access point canbe configured to route the traffic to an appropriate endpoint within adata center providing the service. To do so, the access point may berequired to be aware of a network address of the endpoint. In oneembodiment, each data center is configured with a resource managerdevice that maintains a listing of endpoints providing access to aservice. The resource manager may information the access points of theendpoints, including network address information for the endpoints.Thus, on receiving a request to access a service, each access point maybe configured to select an available endpoint for the service and toroute the request to that endpoint. As will be discussed below, theendpoint may be selected based on a combination of criteria, includingbut not limited to network performance criteria to the endpoint (e.g.,in terms of latency) and desired load balancing across endpoint.

In one embodiment, to route traffic to an endpoint, each access point isconfigured to utilize network address translation (“NAT”). NATtechniques are known within the art, and thus will not be described indetail herein. However, in general, NAT techniques enable a device toact as a middle-man between two devices, while rewriting aspects of eachdata packet, such as a source and destination network address, tofacilitate communications between the devices. In accordance withembodiments of the present disclosure, each access point may operate toreplace a source network address of a requesting device with its ownnetwork address (e.g., a unicast network address uniquely identifyingthe global access point), and to replace the destination network address(e.g., the global network address of a service) with a network addressof an endpoint providing that service. The access point may then routethe packet to the endpoint, receive a response from the endpoint (ifany), perform a reverse translation of source and destination (e.g.,replacing the source of the response with the global network address andthe destination of the response with the network address of therequesting device), and return the packet to the requesting device. Inone embodiment, the access point may utilize port translation (known inthe art) to facilitate distinction of traffic flows (series ofinterrelated packets) when utilizing NAT, to ensure correct translationof addresses when handling traffic from multiple requesting devices.

Beneficially, the use of NAT enables the access point to forward trafficreceived from a client device to an endpoint, with minimal interferenceto a connection between the client device and endpoint. For example, byutilization of NAT, an access point is not required to act as a“termination” point for a certain network protocols, such as theTransmission Control Protocol (TCP). Alternative routing techniques may,for example, cause an access point to accept a TCP connection from aclient device, and initialize a second TCP connection to an endpoint.Maintaining multiple such connections (and correspondences between them)may significantly increase the resource usage of an access point,reducing its capacity to route traffic addressed to global networkaddresses. Thus, use of NAT can provide benefits over such techniques.

Moreover, the use of NAT at an access point enables traffic to bedirected to a global network address different from any address utilizedby an endpoint, thus preventing disclosure of the address actuallyutilized by the endpoint. As such, the endpoint may be protected fromreceiving traffic directly. This is turn may reduce or eliminate thevulnerability of the endpoint to network attacks, such as denial ofservice (DoS) attacks. In this manner, the global access points canprovide benefits similar to those provided by traditional contentdelivery networks (CDNs). However, unlike CDNs, the access points neednot replicate the functionality of service (e.g., by implementing a webserver at each point of presence within the CDN). Rather, the accesspoints of the present disclosure can enable traffic to flow through theaccess point, enabling packets of a requesting device to reach anendpoint providing the service. For this reason among others, the globalaccess points disclosed herein may provide a greater range offunctionality than CDNs, enabling load balancing and distribution amongany number of different network-based services.

In another embodiment, to route traffic to an endpoint, each accesspoint is configured to utilize encapsulation. Encapsulation is a knownnetworking technique, and thus will not be described in detail herein.However, in general encapsulation can be utilized to add additionalinformation (frequently in the form of a header, and sometimes also inthe form of a trailer) to a data packet, thus “wrapping” orencapsulating the data packet to result in an encapsulated data packet.In the context of the present disclosure, encapsulation may be utilizedto provide a “network tunnel” between each access point and endpoint.Specifically, when a packet is received at an access point from a clientdevice and addressed to a network address of a service provided by anendpoint, the access point may encapsulate the packet with additionalinformation enabling the packet to be routed to a selected endpoint,such as a network address of the endpoint. The encapsulated packet maythen be routed to the endpoint over a physical network. The endpoint may“decapsulate” the encapsulated packet to remove the additionalinformation, and process the decapsulated packet as if it were receiveddirectly from the client device.

In one embodiment, the endpoint may respond directly to the clientdevice, via a physical network connecting the endpoint to the clientdevice. In another embodiment, the endpoint may respond to the clientdevice by encapsulating a response packet, and transmitting theencapsulated response packet back to the access point. The access pointmay in turn decapsulate the response packet and return the responsepacket to the client. Direct return of responses from an endpoint to aclient device may beneficially reduce workload of an access point, andmay also reduce traffic on a physical network connecting the accesspoint to the endpoint. However, direct return of responses may berequired to utilize a physical network connecting the endpoint to theclient device, which may not be as robust as the physical networkconnecting the access point to the endpoint. For example, where theendpoint and access point are connected via a private network, while theaccess point, endpoint, and client device are connected via a publicnetwork (such as the Internet), return of responses through the accesspoint (rather than direct return) may be preferable, as it may bepreferable to minimize the total distance traveled on the public network(e.g., where the access point is located closer to the client devicethan the endpoint).

In general, encapsulation of packets between access points and endpointsmay increase the computational resource usage of access points andendpoints, as compared to use of NAT to route packets from access pointsto endpoints. However, encapsulation may also provide benefits over aNAT implementation. For example, encapsulation may provide forresiliency of TCP session in the event that packets of a client deviceare routed to different global access points during the TCP session.Such routing may occur, for example, based on operation of networksbetween the client device and the different access points, due topotential changes in how the networks process anycast advertisements ofthe different access points. Where an access point utilizes NAT to routepackets to an endpoint, each access point may maintain NAT informationindependently. Thus, if client device packets are routed to a differentaccess point, that second access point may not have sufficientinformation to gracefully take over communications of the client (e.g.,the transformed packets of the second access point may not be identicalto what would have been generated at the initial access point). This mayresult in the TCP session of the client device and the endpoint beingbroken, requiring that the client device reestablish the TCP connection.

In contrast, encapsulation can enable graceful handling of rerouting ofclient requests between access points. Under either NAT or encapsulationimplementations, each access point may be configured to select anendpoint in a consistent manner (e.g., applying the same selectionalgorithm). Thus, client device packets would be expected to be routedto the same endpoint, regardless of the access point to which they weresent initially. Moreover, by utilizing encapsulation, no data of theinitial client device's packets need be changed when routing the packetsto the endpoint. Rather, that data can be encapsulated (e.g., withadditional information enabling routing to the endpoint), and regainedat the endpoint after decapsulation. For this reason, even if clientpackets are routed to different access points, the final data availableto the endpoint after decapsulation can be the same, enabling theendpoint to maintain a TCP session with the client device even when dataflows through different access points.

In some embodiments, to further improve performance in implementationsthat utilize encapsulation, each access point can be configured toassist endpoints in establishing connection-oriented communicationsessions, such as TCP sessions, with client devices. Connection-orientedcommunication sessions generally require a session initialization phase,where the two parties to the session communicate to establish mutualunderstanding of communication state. TCP sessions, for example, utilizea “three-way handshake.” The TCP three-way handshake is known in the artand thus will not be described in detail herein. However, in brief, aTCP session is established when a first party sends a synchronize(“SYN”) packet to a second party (including a first party sequencenumber to be used during the session), the second party responds with asynchronize-acknowledge (“SYN-ACK”) packet (acknowledging the firstparty sequence number and including a second party sequence number), andthe first party responds to the SYN-ACK packet with an acknowledgement(“ACK”) packet (acknowledging the second party sequence number). Becausethe three-way handshake requires three separate communications betweenthe parties, increases in latency between the parties during thehandshake are magnified three-fold. In the context of the presentdisclosure, for example, if communications between a client device andendpoint incur a 100 millisecond (ms) latency, the three-way handshakecan be expected to take at least 300 milliseconds. Since such ahandshake is required prior to transmission of data over a TCP session,it is beneficial to reduce the time required to initialize the TCPsession.

Accordingly, in embodiments of the present disclosure, each globalaccess point may be configured to conduct a TCP three-way handshake witha client device, and then to “hand off” the TCP connection to anendpoint. Specifically, each access point may be configured to “listen”for incoming SYN packets from client devices, and to respond to suchpackets by conducting the TCP three-way handshake. After the handshakeis completed, the access point may transfer TCP session information(e.g., the first and second party sequence numbers of the TCP session)to an endpoint selected to provide a requested service to the clientdevice. On receiving the TCP session information, the endpoint may“adopt” the TCP session, and process client device data packets as ifthe TCP session had been established between the client device and theendpoint. Because the access point can be assumed to be “closer” to theclient device in terms of latency, the time required to establish theTCP session is reduced, without interfering with the ability of theclient device and the endpoint to communicate via a common TCP session.While examples are provided herein with respect to TCP handshakes,statefully communication sessions each generally require aninitialization phase. Embodiments described herein may be utilized toconduct such an initialization phase at an access point, while handingoff context for the connection-oriented communication session to anendpoint to enable a client device and the endpoint to communicate viathe connection-oriented communication session.

As will be appreciated by one of skill in the art in light of thepresent disclosure, the embodiments disclosed herein improves theability of computing systems to provide network-accessible services.Specifically, embodiments of the present disclosure improve on priorload balancing techniques, by providing scalable, resilient, andresponsive load-balancing across a common network address, improving onknown load balancing techniques. Moreover, the presently disclosedembodiments address technical problems inherent within computingsystems; specifically, the limited nature of computing resources withwhich to provide network-accessible services and the difficulties ofload-balancing requests to such services in a scalable, resilient, andresponsive manner. These technical problems are addressed by the varioustechnical solutions described herein, including the use of a distributedset of access points associated with a common network address, eachconfigured to receive requests for a service, and to route the requeststo endpoints of the service based at least partly on load-balancing therequests among the endpoints. Thus, the present disclosure represents animprovement on existing network load-balancing systems and computingsystems in general.

The foregoing aspects and many of the attendant advantages of thisdisclosure will become more readily appreciated as the same becomebetter understood by reference to the following description, when takenin conjunction with the accompanying drawings.

FIG. 1 is a block diagram depicting an illustrative logical environment100 including multiple client devices 102 in communication with a set ofglobal access points 106A-N via a first network 104, which global accesspoints 106A-N are in communication with a set of data centers 110A-N viaa second network 108. While the client devices 102, global access points106 and data centers 110 are within FIG. 1 in groups, the client devices102, global access points 106 and data centers 110 may be geographicallydistant, and independently owned or operated. For example, the clientdevices 102 could represent a multitude of users in various global,continental, or regional locations accessing network-accessible servicesprovided by the data centers 110, which data centers may further bedistributed among various global, continental, or regional locations.The global access points 106 may similarly be distributed. In oneembodiment, the data centers 110 represent devices in locations undercontrol of a single entity, such as a “cloud computing” provider, whilethe global access points 106 represent devices in co-tenanted locations,such as network “points of presence” or Internet Exchange Points (IXPs).The global access points 106 may generally be more numerous than thedata centers 110 and in distinct physical locations. However, in otherembodiments, one or more of the access points 106 may be located withinone or more data centers 110. Accordingly, the groupings of clientdevices 102, access points 106, and data centers 110 within FIG. 1 isintended to represent a logical, rather than physical, grouping.

The networks 104 and 108 may be any wired networks, wireless networks orcombination thereof. In addition, the networks 104 and 108 may be apersonal area network, local area network, wide area network, cablenetwork, satellite network, cellular telephone network, or combinationthereof. In the example environment of FIG. 1, network 104 is a globalarea network (GAN), such as the Internet, while the network 108 is aprivate network dedicated to traffic associated with an entity providingthe data centers 110 and access points 106. Protocols and components forcommunicating via the other aforementioned types of communicationnetworks are well known to those skilled in the art of computercommunications and thus, need not be described in more detail herein.

While each of the client devices 102 and access points 106 are depictedas having a single connection to the network 104, individual componentsof the client devices 102 and access points 106 may be connected to thenetwork 104 at disparate points (e.g., through different neighboringnetworks within the network 104). In some embodiments, the data centers110 may additionally or alternative be connected to the network 104.Similarly, while each of the access points 106 and data centers 110 aredepicted as having a single connection to the network 108, individualcomponents of the access points 106 and data centers 110 may beconnected to the network 108 at disparate points. Accordingly,communication times and capabilities may vary between the components ofFIG. 1. The network configuration of FIG. 1 is intended to beillustrative of a communication path in embodiments of the presentdisclosure, and not necessarily to depict all possible communicationspaths.

Client devices 102 may include any number of different computing devicescapable of communicating with the global access points 106. For example,individual client devices 102 may correspond to a laptop or tabletcomputer, personal computer, wearable computer, server, personal digitalassistant (PDA), hybrid PDA/mobile phone, mobile phone, electronic bookreader, set-top box, camera, digital media player, and the like. In someinstances, client devices 102 are operated by end users. In otherinstance, client devices 102 themselves provide network-accessibleservices, which interact with the global access points 106 to accessother network-accessible services.

The data centers 110 of FIG. 1 illustratively include endpoint computingdevices providing one or more network-accessible services on behalf ofone or more service providers. Illustratively, the data centers 110 maybe operated by a “cloud computing” provider, which makes host computingdevices within the data center available to service providers forproviding their services. The cloud computing providing may generallymanage operation of the data center, while providing various mechanismsfor the server providers to configure their respective endpoints. Oneillustrative configuration of a data center 110 is provided below withrespect to FIG. 2.

In accordance with embodiments of the present disclosure, the cloudcomputing provider may enable service providers to associate theirendpoints with one or more global network addresses, which areaddressable on the network 104 to interact with the data centers 110 ina load-balanced manner. The cloud computing provider may further enablethe service providers to specify how such load-balancing should occur,such as by specifying a percentage of requests to be routed to each datacenter 110. The cloud computing provider may further enable the serviceproviders to alter the configuration of endpoints independently of theglobal network addresses, such that altering the specific endpointsproviding a service does not require reconfiguration of the networkaddresses. Use of global network addresses may significantly simplifyoperation of network services, since any client device 102 wishing toconnect to the service may simply transmit a request to a global networkaddress of the service. Alterations to the endpoints providing theservice may then be made without the need to alter DNS records for theservice, for example. As will be described below, in some instancesthese alterations may be made automatically, such that no user action isneeded even as endpoints for the service change within the data centers110.

To facilitate global network addresses, a set of global access points106A-N are provided. Each access point may generally include one or morecomputing devices configured to obtain requests from client devices 102to interact with services, and to route such requests to an endpointwithin a data center 110 selected based at least partly onload-balancing requests across the data centers 110. Access points 106may further act as a type of proxy for the endpoints, enabling trafficbetween client devices 102 and data centers 110 to flow across theaccess points 106. Operation of access points 106 is discussed in moredetail below. However, in brief, the may utilize anycast techniques tobroadcast availability of global network addresses to neighboringnetwork devices within the network 104, which in one embodiment includesdevices not under the control of a common entity as provides the accesspoints 106A. The access points 106 may thus attract traffic addressed tothe global network addresses. The access points 106 may thereafterselect an endpoint to which to direct the traffic, based on factors suchas availability of endpoints, load-balancing across data centers 110,and performance criteria between the access point 106 and the variousdata centers 110.

After selecting a data center 110, an access point 106 can route therequest to the endpoint. In one embodiment, the access point 106 usesNAT translation or encapsulation to redirect the request to the endpointover the network 108, preventing disclosure of a network address of theendpoint to the client devices 102. Where connection-orientedcommunication sessions are utilized between client devices 102 and anendpoint, the access point 106 may operate to conduct an initializationphase of the communication session on behalf of the endpoint, inaccordance with the present embodiments. In instances where the network108 is a private network, the global access points 106 may furtherfunction as an “offloading” point for traffic to the endpoints, movingthat traffic from a public network (e.g., network 104) to the privatenetwork 108. Generally, such a private network would be expected to havegreater performance than a public network, and thus such offloading mayfurther increase the speed of communication between client devices 102and endpoints.

As noted above, the access points 106 may implement a variety oftechniques to ensure resiliency of a network service using a globalnetwork address. Illustratively, the use of anycast to advertise accesspoints 106 may provide resiliency between access points 106, as thefailure of an individual access point 106 can generally be expected tocause devices of the network 104 to route requests to another accesspoint 106. Moreover, to address potential failures of the network 104,each access point 106 can be configured to control its announcement ofglobal network addresses on the network 104, providing multiple routingpaths for each service to the access point 106. Additional detailsregarding such control of announcements to provide routing paths isprovided below. Still further, to address potential failures within anaccess point 106, each access point 106 may be configured to includemultiple flow managers, handling different traffic flows addressed toglobal network addresses. The flow managers may be distributedlogically, such as across program threads, and/or physically, such asacross processors or computing devices. Thus, failure of one flowmanager may have little or no impact on other flow managers within anaccess point 106, limiting impact of partial failures within an accesspoint 106. One illustrative configuration of an access point 106 isdiscussed below with reference to FIG. 4.

FIG. 2 is a block diagram depicting an illustrative configuration of adata center 110 of FIG. 1. As shown in FIG. 2, the data center 110includes an endpoint pool 201 containing a set of endpoints 202A-N. Eachendpoint 202 illustratively represents a computing device configured toprovide access to a network-accessible service. In one embodiment,endpoints 202 are individual physical computing devices. In anotherembodiment, endpoints 202 are virtualized computing devices executing onphysical computing devices. In yet another embodiment, endpoints 202 arecollections of computing devices (physical or virtualized) collectivelyconfigured to provide access to a network-accessible service. Forexample, each endpoint 202 may be a collection of devices being a loadbalancer device configured to load balance requests to the endpoint 202among the collection of devices. Each endpoint 202 is in communicationwith the network 108, and thus addressable on the network 108. Thenumber of endpoints 202 may vary, for example, depending on the capacityrequirements of the network-accessible service. Illustratively, aservice provider for such service may contract with an operator of thedata center 110 (e.g., a cloud computing provider) to generate andprovision the endpoints 202.

In one embodiment, the number of endpoints 202 may vary according to apresent or historical demand for the network-accessible service. Tofacilitate a varying number of endpoints 202, the data center mayinclude a resource manager 206 (e.g., implemented directly on a physicalcomputing device or as a virtualized device on a host physical computingdevice) that monitors load of the endpoints 202 (e.g., with respect torequests per second, computing resource usage, response time, etc.), anddynamically adjusts the number of endpoints 202 to maintain load withinthreshold values. For example, where the endpoints 202 are implementedas virtualized devices, the resource manager 206 may generate andprovision new virtualized devices when a current set of endpoints 202have usage metrics that exceed desired upper usage parameters, and“spin-down” and remove virtualized devices when metrics fall below lowerusage parameters. The resource manager 206 may further be configured, onmodifying a number of endpoints 202 within the pool 201, to notify theglobal access points 106 of network address information for endpoints202 within the pool 201, such that the access points 106 may addresstraffic to the endpoints 202.

In addition, the data center 110 of FIG. 2 includes a health checkdevice 204, configured to monitor the health of endpoints 202 within thepool 201. Illustratively, the health check device 204 may periodically(e.g., every n seconds) transmit a request to the endpoints 202, andmonitor for whether an appropriate response is received. If so, thehealth check device 204 may consider the endpoint 202 healthy. If not,the health check device 204 may consider the endpoint 202 unhealthy. Thehealth check device 204 may illustratively notify the global accesspoints 106 of unhealthy endpoints 202 to cause the access points 106 toreduce or eliminate routing of traffic to the endpoint 202 while theendpoint 202 is unhealthy. In some instances, the health check device204 may further be configured to check the health of the global accesspoints 106, the health of endpoints 202 in other data centers 110, orthe health of a network path between the health check device 204 and theaccess points 106. For example, the health check device 204 mayperiodically transmit information to the access point 106, and monitor aresponse from the access point 106, network metrics regarding theresponse (e.g., latency), or the like. The health check device 204 mayreport this information to the access points 106 to facilitate routingof traffic to endpoints 202A within the data center 110. Illustratively,the health check device 204 may report health information for endpoints202 to the configuration data store 210 (e.g., a database on the datastore 210), which may be propagated to the access points 106 byoperation of the configuration manager 208 in the manner described below(e.g., as part of a configuration package or parcel).

The data center 110 of FIG. 2 further includes a configuration manager208, configured to enable service providers to configure operation ofthe data centers 110 and global access points 106. Illustratively, theconfiguration manager 208 may provide an interface through which usersmay specify endpoints 202 that provide a network-accessible service,configure those endpoints 202 and configure the resource manager 206 toscale up or down endpoints. The configuration manager 208 may furtherenable service providers to assign global network address to thoseendpoints, and to specify load-balancing parameters for routing trafficaddressed to global network address to various data centers 110. Theconfigurations created by service providers may be stored within aconfiguration data store 210, which can corresponds to any persistent orsubstantially persistent storage device (e.g., hard disk drives, solidstate drives, network-attached storage devices, etc.). In someinstances, the configuration data store 210 may include multiplerepresentations of a configuration of a service provider. For example,to facilitate rapid reconfiguration of global access points 106, theconfiguration data store 210 may include a database (such as arelational database) that is modified each time a service providercommits a change to their configuration. The configuration manager 208may periodically (e.g., each 100 milliseconds, 1 second, 2 seconds, 5seconds, 30 seconds, etc.) determine whether changes have been made tothe database, and if so, generate a new configuration package for theglobal access points 106, which configuration package encompasses thechanges to the database (and thus, service provider's configuration)relative to a prior configuration package. The configuration manager 208may then store the configuration package into the configuration datastore 210 for retrieval by the global access points 106. In oneembodiment, each global access point 106 is configured to periodically(e.g., each 100 milliseconds, 1 second, 2 seconds, 5 seconds, 30seconds, etc.) poll the configuration data store 210 to determinewhether a new configuration package exists, and if so, to retrieve andimplement the package. In some instances, a configuration package may bedivided into package “parcels,” representing a portion of theconfiguration. Global access points 106 may be configured to retrieveonly those parcels modified with respect to an existing parcel.Modifications may be tracked, for example, based on versioning ofparcels or a package. Still further, in some embodiments, packages orparcels may be stored in the data store 210 as differences or “deltas”from a prior version, such that an access point 106 may retrieve onlychanges since a prior version of the parcel, reducing the data transferrequired to update a package or parcel. In one embodiment, theconfiguration manager 208 may periodically (e.g., each 100 milliseconds,500 milliseconds, etc.) “checkpoint” packages or parcels, by collectingall changes since a prior checkpoint and storing the package or parcelas a standalone version. Such checkpointing may facilitate rapidreconfiguration in the instance that a global access point 106 has noframe of reference of a prior package or parcel.

In one embodiment, only one data center 110 of the data centers 110A-Nof FIG. 1 includes the configuration manager 208, which manager 208propagates each service provider's configuration to the other datacenters 110 and access points 106. In another embodiment, each datacenter 110 includes a configuration manager 208, and the managers 208 ofeach data center 110 may communicate to synchronize configurationsbetween the data centers 110 and with the access points 106.

While only some components of the data center 110 are shown as incommunication with the network 108, other components may additionally bein communication with the network 108 and/or the network 104. The linesof FIG. 2 are not intended to represent all actual or potential networkconnections, but rather to illustrate a possible flow of service-relatedtraffic to endpoints 202.

Moreover, while shown within a data center 110, in one embodiment,global access points 106 may also include a configuration manager 208,enabling configuration of the access point 106 directly. In anotherembodiment, the global access points 106 exclude any configurationmanager 208 and data store 210. For example, where access points 106 areimplemented at co-tenanted environments (e.g., not operated by oraccessible to parties other than an operator of the access points 106),the access points 106 may be configured to exclude any persistentstorage, and to instead retrieve configuration information from a datacenter 110 on initialization of the access point 106. In this manner,security of the access points 106 may be increased, as powering down theaccess point 106 would be expected to result in loss of any sensitivedata that may reside on the access point 106.

While the data center 110 is shown as including one endpoint pool 201,corresponding to one network-accessible service, the data center 110 mayhost numerous pool 201, each corresponding to a different service. Thus,multiple service providers may utilize a data center 110. Moreover, asnoted above, each network-accessible service may be provided byendpoints 202 across multiple data centers 110. Accordingly, the globalaccess points of FIG. 1 may distribute traffic to such a service acrossthe data centers 110.

In accordance with embodiments of the present disclosure, the datacenter 110 further includes a session handoff manager 212, configured tofacilitate adoption of connection-oriented communication sessions withclients 102 initialized at a global access point 106. As discussedabove, global access points 106 may in some instances be configured tocomplete an initialization phase of a connection-oriented communicationsession, such as the TCP three-way handshake, and to thereafter hand offthe communication session to an endpoint 202. In some instances, anendpoint 202 may be configured to accept such a hand off, by receivingcontext of the connection-oriented communication session (e.g., sequencenumbers of both ends of the connection) and generating local stateinformation incorporating that context. The endpoint 202 may be soconfigured, for example, by modification of networking protocol handlerswithin an operating system of an endpoint 202 (e.g., by modification ofa kernel to accept and adopt TCP context information from a globalaccess point 106). However, it may be preferable that endpoints 202 arenot required to be so configured, in order to enable a wide variety ofendpoints 202 to utilize global access points 106. To enable this, thedata center 110 may also include a session handoff manager 212, which isconfigured to accept and adopt a connection-oriented communicationsession from a global access point 106. The session handoff manager 212may then establish a separate connection-oriented communication sessionwith the endpoint 202 selected by the access point 106 to provide aclient device 102 with a service, and operate as a “middle man” betweenthe two sessions. Because the session handoff manager 212 and theendpoint 202 can generally be co-housed within the data center 110,creation of a second connection-oriented communication session can beexpected to inject minimal delay into communications between the clientdevice 102 and the endpoint 202.

FIG. 3 is a block diagram depicting an illustrative configuration of aglobal access point of FIG. 1. As shown in in FIG. 3, each global accesspoint 106 is in communication with the network 104 via a router 302.While only a single router 302 is shown in FIG. 2, access points 106 mayinclude multiple routers 302. Moreover, while a single connection to thenetwork 104 is shown, each router 302 may include multiple connectionsto the network 104, potentially to multiple different neighboringdevices within the network 104, each of which may correspond todifferent sub-networks (e.g., autonomous systems (AS's) within thenetwork 104).

As noted above, global access points 106 may be configured to utilizeanycast techniques to attract traffic to global network addressesassociated with network-accessible services. As such, the router 302 isillustratively configured to advertise the global network addresses toneighboring devices on the network 104. In one embodiment, suchadvertisements are BGP advertisements. Such advertisements can cause therouter 302 to attract traffic addressed to the global network addresses,as the advertisements can cause devices on the network 104 to routetraffic addressed the addresses to the router 302, in accordance withoperation of anycast techniques.

As discussed above, the global access point 106 may implement a varietyof techniques to increase resiliency of the access point 106. In oneembodiment, the global network addresses advertised by the access point106 are divided into multiple address groups. To decrease the potentialeffects of failures on the network 104, the router 302 (or multiplerouters 302) can be configured to transmit BGP announcements for eachaddress group to different neighboring devices on the network 104 (e.g.,different AS's). A network-accessible service may be associated withaddresses from multiple address groups, each of which may be provided toclient devices 102 as an address at which to access the service. Becauseaddress from different groups are advertised differently on the network104, different routing paths can be expected on the network 104 foraddresses of each group. For example, packets addressed to addresseswithin a first group may reach the router 302 over a first AS of thenetwork 104, while packets addressed to addresses within a second groupmay reach the router 302 over a second AS. Thus, if a failure were tooccur within the first AS (or a downstream AS connected to the firstAS), packets addressed to addresses within the second group may beexpected to still reach the router 302, and vice versa. As such,dividing global network addresses into multiple groups can increaseresiliency of the access points 106 to failures within the network 104.

On receiving a packet addressed to a global network address, the router302 may route the packet to a flow manager 304, from a set of flowmanagers 304A-N. While an access point 106 may implement a single flowmanager 304, it may be beneficial for an access point to implementmultiple flow managers 304 to provide redundant operation of such flowmanagers 304. The router 302 may use any number of known load-balancingtechniques to distribute packets to the flow managers 304, such as roundrobin selection. In one embodiment, the router 302 utilizes consistenthashing to distribute packets. Consistent hashing is known in the artand will thus not be described in detail herein. Consistent hashing maybe beneficial, for example, in increasing the changes that multiplepackets with the same characteristics (e.g., source network address,source network port, destination network address, destination networkport, protocol) are routed to the same flow manager 304. This maybeneficially enable the flow managers 304 to maintain state informationregarding flows of packets between a client device 102 and a destinationendpoint 202. In some instances, such state information may be required,for example, to implement NAT techniques, or to conduct aninitialization phase of a connection-oriented communication session. Inanother embodiment equal-cost multi-path (ECMP) load balancing is usedto route traffic to the flow managers 304A-N. ECMP load balancing isknown in the art and thus will not be described in detail herein.

In one embodiment, ECMP load balancing is applied to route packets tothe flow managers 304A based on a global network address to which thepacket is addressed. Illustratively, each flow manager may handlepackets addressed to a subset of addresses within a group of globalnetwork addresses. For example, a first flow manager 304 may preferablyhandle a first quartile of addresses within a group, a second manager304 may preferably handle a second quartile, etc. By dividing networkaddresses within a group among different flow managers 304, the accesspoint 106 may for example decrease the portion of services that areadversely effected by improper operation of a flow manager 304, such asdue to a misconfiguration of a service associated with an addresshandled by that flow manager 304. Where multiple groups of globalnetwork addresses are used, services may be “shuffled” between differentgroups, such that two services having addresses sharing a subset underone address group are unlikely to have addresses sharing a subset underanother address group. Such shuffling can reduce the total percentage ofservices made completely unavailable from an access point 106 due tomalfunction of an individual flow manager 304A.

To further facilitate use of state information on flow managers 304, therouter 302 in one embodiment implements a flow hash routing algorithm,or similar algorithm to identify related packets constituting a “flow”of traffic between a client 102 and an endpoint 202. A variety of flowidentification algorithms are known in the art, and thus will not bedescribed in detail herein. Illustratively, the router 302 may implementa flow identification algorithm to recognize flows of data packets, andconsistently route packets of the same flow to the same flow manager304. In one embodiment, the router 302 applies flow-based routing to adata packet prior to otherwise distributing packets to the flow managers304, such that if the packet is recognized as part of a flow, it isdistributed to the same flow manager 304 that previously handled packetsof that flow. In such an embodiment, if the packet is not recognized aspart of the flow, additional load-balancing algorithms (e.g., consistenthashing) are applied to the packet to determine a flow manager 304 towhich to route the packet.

On receiving a data packet, a flow manager 304 may determine a datacenter 110 to which to route the packet, as well as an endpoint 202within the data center 110. In one embodiment, the flow manager 304 mayapply a combination of criteria to select a data center 110 to which toroute a packet, including network performance criteria andload-balancing criteria. For example, a flow manager 304 may, for agiven packet, initially select a data center 110 at a minimum networkdistance (e.g., across the network 108) from the access point 106.Network performance criteria may include, for example, latency, numberof hops, bandwidth, or a combination thereof. In general, routing of apacket to a data center 110 with a maximum network performance criteriamay beneficially increase the speed of communication between a clientdevice 102 and the data center 110. However, because network performancecriteria is unlikely to rapidly shift between an access point 106 and adata center 110, simply routing each packet to a data center 110 with amaximum expected performance criteria may not achieve the load balancingrequested by a service provider. As such, each flow manager 304 mayfurther modify an initially selected data center 110, as necessary toachieve the desired load balancing of a service provider. For example,where a service provider desires a maximum of 70% of traffic to berouted to a data center 110 that is closest to an access point 106 interms of minimized latency or network distance (e.g., number of hops), aflow manager 304 may be configured to route a first 7 packets in a giventime period (e.g., each second) to the data center 110, and to route thenext 3 packets to an alternative data center 110 (e.g., a second closestdata center 110 in terms of network distance, latency, etc.). Whilepercentages are discussed herein as an example load balancing criteria,additional or alternative criteria may be used. For example, a serviceprovider may specify a cap in the total number of packets to be routedto a data center 110.

In another embodiment, rather than initially selecting a data center 110based on performance criteria between the access point 106 and the datacenter 110, global access points 106 may be configured to initiallyselect a data center 110 based on performance criteria between theclient device 102 from which traffic was received and the initiallyselected data center 110. Beneficially, use of network performancecriteria between client devices 102 and data centers 110 may result inconsistent selection of a data center 110 for a given client device 102,regardless of the access point 106 to which the client device's trafficis routed. After selection of an initial data center 110 (e.g., closestto a client device 102 in terms of distance or minimized latency), theaccess point 106 may modify that selection as required to achieveload-balancing specified by a service provider. For example, if theservice provider desired no more than 70% of traffic to be routed to theinitially selected data center 110, and the proportion of traffic over apast period of time exceeds that percentage, the access point 106 mayselect a different data center (e.g., a next most performant data center110 with respect to the client device 102). The access point 106 maythen determine whether the request could be routed to that differentdata center 110, based on a proportion of historical traffic routed tothat different data center 110. This process may continue until theaccess point 106 locates a data center 110 that is acceptable based onload-balancing specifications of the service provider.

In some embodiments, load balancing is implemented locally at each flowmanager 304. In other embodiments, load balancing is implemented acrossall flow managers 304 of an access point 106. In still otherembodiments, load balancing is implemented across flow managers 304 ofmultiple access points 106. In general, localized load balancing isexpected to be less resource intensive, as it requires lesscommunication between distributed components. However, less localizedload balancing may result in load balancing criteria more closelyresembling that desired by a service provider.

In some instances, flow managers 304 may implement a hybrid of localizedand non-localized load balancing. For example, each flow manager 304 mayimplement localized load balancing (e.g., localized to each manager 304or each access point 106), and periodically negotiate with other accesspoints 106 to adjust the weights applied when selecting a data center110. For example, where a service provider requests that traffic bedivided evenly between data centers 110, localized load balancing maycause each of two access points 106 to divide traffic evenly between thedata centers 110. This may result in less than optimal routing, as halfof traffic at each access point 106 may be routed to a non-closest datacenter 110. Accordingly, in this scenario the access points 106 maycommunicate regarding their routing of traffic, and assuming forhypothetical sake that the volume of traffic at each point 106 is equal(and that only two access points 106 are considered), each access point106 may begin to route all of their packets to a nearest data center110. Such a division would still result in even division of trafficamong the data centers 110, and moreover beneficially increase theaverage network performance metric for each flow of packets.

In one embodiment, a desired proportion or volume of traffic routed to agiven data center 110 may be statically selected by a service provider.For example, a service provider may request that no more than 70% ofrequests at an access point 106 be routed to a given data center 110. Inanother embodiment, the desired proportion or volume may be dynamic. Forexample, a service provider may specify that a desired proportion orvolume of traffic routed to a given data center 110 increase or decreasefrom a first point in time to a second point in time, according to agiven rate of change (e.g., linearly, exponentially, logarithmically,etc.).

After selecting a data center 110 to which to route traffic, a flowmanager 304 may select an endpoint 202 within the data center 110 towhich to route the traffic. The endpoint 202 may be selected accordingto any load balancing algorithm. In one embodiment, the flow manager 304may utilize consistent hashing to select an endpoint 202.

Similarly to as discussed above with respect to the flow managers 304,it may be desirable for traffic of a given flow to be consistentlyrouted to the same endpoint 202, such that the endpoint 202 may maintainstate information regarding that flow. As such, each flow manager 304may utilize a flow detection algorithm (similarly to as discussed abovewith respect the router 302) to detect subsequent packets within a flowpreviously routed by the flow manager 302A to an endpoint 202, and toroute such packets to the same endpoint 202. In one embodiment, where apacket is identified by the flow managers 304 as part of analready-routed flow, the flow manager 304 may omit selection of a datacenter 110 and endpoint 202 for the packet, in favor of routing thepacket to the endpoint 202 previously used for prior packets of theflow.

After selecting an endpoint 202 to which to route a packet, a flowmanager 304 can modify the packet to facilitate routing to the endpoint202. For example, a destination network address of the packet, whenreceived at the router 302, may be a global network address. The flowmanager 304 may therefore modify the packet to replace the destinationnetwork address with a network address of the endpoint 202. In oneembodiment, each flow manager 304 implements NAT techniques to modifypackets addressed to global network addresses. For example, each flowmanager 304 may, for packets bound for endpoints 202, replace a globalnetwork address with a network address of an endpoint 202 as thedestination network address of the packet, and replace a network addressof the client device 102 with an address of access point 106 as a sourceaddress. Similar translation may occur for packets from endpoints 202 tobe routed to client devices 102, in accordance with NAT techniques. Flowmanagers 304 may illustratively use port translation (a known NATtechnique) to distinguish between translated flows. After translation,the flow manager 304 can return the packet to the router 302 fortransmission to the selected endpoint 202 over the network 108.

In another embodiment, a flow manager 304 may utilize encapsulation toroute a packet to an endpoint 202. Illustratively, each flow manager 304may generate an IP “tunnel” to a device within a data center 110, suchas the session handoff manager 212 or a router within the data center110. To route packets to an endpoint 202, a flow manager 304 mayencapsulate the packet, and transmit the packet to the receiving devicevia the tunnel. The receiving device may then decapsulate the packet andtransmit the packet to the endpoint 202. In one embodiment, the flowmanager 304 replaces a destination address of the packet (e.g., a globalnetwork address of the service) with a destination address of theselected endpoint 202 to facilitate transmission of the packet 202 tothe endpoint. Encapsulation of packets over a tunnel may providebenefits, such as preservation of the network address of a client device102 that transmitted the packet. As such, encapsulation may enable, forexample, a client device 102 and an endpoint 202 to preserveconnection-oriented communication sessions even when packets of a clientdevice 102 are routed through different access points 106. In someinstances, flow managers 304 and endpoints 202 (or session handoffmanagers 212) may be configured to authenticate communications betweenone another, to ensure authenticity of packets transmitted between thedevices. For example, each of the flow managers 304 and endpoints 202(or session handoff managers 212) may be configured to generate adigital signature authenticating the respective devices, and to includesuch a digital signature in a header of encapsulated packets flowingbetween the devices, such that each device can authenticate the digitalsignature to verify that packets were generated at a known device. Asanother example, the flow managers 304 and endpoints 202 (or sessionhandoff managers 212) may be configured to communicate via known secureprotocols, such as the Transport Layer Security (TLS) protocols.

As noted above, access points 106 may in some instances be configured toconduct an initialization phase of a connection-oriented communicationsession, to reduce the time needed to conduct such a phase (e.g., due tothe relative proximity of the access point 106 to the client device 102relative to the data center 110). To facilitate such functionality, eachflow manager 304 can implement the initialization phase with a client,and provide information related to an initialized session to a devicewithin the data center 110, such as an endpoint 202 or a session handoffmanager 212. The information may be provided, for example, via thetunnel. Illustratively, a flow manager 304 may generate a tunnel to thesession handoff manager 212, and pass within the tunnel a “TCP handoff”command that includes TCP session state information. The stateinformation may include, for example, a TCP “five-tuple” (five valuesthat often define a TCP connection: source address, source port,destination address, destination port, and the protocol in use) or aportion thereof, and/or sequence numbers of the TCP connection. Onreceiving the TCP handoff command, the session handoff manager 212 maygenerate a corresponding entry in its own stateful session table (e.g.,TCP table), thus “adopting” the connection-oriented communicationsession. The flow manager 304 may then transmit packets from the clientdevice 102 via the tunnel, which may be decapsulated and processed atthe session handoff manager 212 as part of the adopted session. Inembodiments where a session handoff manager 212 is used to accomplishhandoff of stateful sessions, flow managers 304 may not be required toselect an endpoint 202 to which to transmit a packet. Rather, flowmanagers 304 may be configured to consistently select an appropriatesession handoff manager 212 within a selected data center 110 as adestination for client device 102 packets. The session handoff manager212, in turn, may select an endpoint 202 within the data center 110.Thus, use of a session handoff manager 212 may shift responsibility forselecting an endpoint 202 from the flow manager 304 to the sessionhandoff manager 212. Selection of an endpoint 202 at a session handoffmanager 212 may occur similarly to such selection by a flow manager 304.

In addition to one or more routers 302 and flow managers 304, the globalaccess point 106 of FIG. 2 includes a health check device 306. Thehealth check device 306 may illustratively transmit health checks toflow managers 304 in order to ascertain whether such flow managers 304are malfunctioning. Transmission of health check data may occursimilarly to transmission of health check data by a health check device204 within a data center 110, as discussed above. Additionally oralternatively, the health check device 306 may transmit health checkdata to endpoints 202 and/or client devices 102. Health check datareceived with respect to endpoints 202 may facilitate routing decisionsto the endpoint 202. For example, if the health check device 306 isunable to communicate with an endpoint 202, a flow manager 304 may haltselection of the endpoint 202 as a destination for traffic, regardlessof whether a health check device 204 local to the endpoint 202 reportsthe endpoint 202 as healthy. Health check data collected with respect toclient devices 102 may illustratively be used to modify availabilityannouncements (e.g., BGP announcements) to devices of the network 104.For example, an operator of the global access points 106 may reviewclient device 102 related health check data to adjust announcements inan attempt to redirect traffic from some devices 102 to a different(e.g., closer) access point 106 than it is currently routed to on thenetwork 104.

The global access point 106 of FIG. 3 further includes a configurationmanager 308 configured to receive configuration information related toservices associated with global network addresses, and to configureoperation of the global access point 106 to implement suchconfiguration. For example, the configuration manager 308 may receiveinformation as to how the router 302 should advertise global networkaddresses to the network 104, information mapping services madeavailable at the data centers 110 to global network addresses routed bythe access points 106, information identifying data centers 110 andendpoints 202 of such services, information specifying a desiredload-balancing among such data centers 110, and the like. In oneembodiment, the configuration manager 308 retrieves configurationinformation from a configuration data store 210 within a data center110. For example, the configuration manager 308 may periodically pollthe data store 210 for new configuration information. As anotherexample, a data center 110 may “push” changes in configuration to theconfiguration manager 308, utilizing a variety of known pushnotification techniques. For example, a data center 110 may implement apublish-subscribe (“pub/sub”) messaging system, and a configurationmanager 208 may publish changes to a service provider's configuration tothe system. The system may then notify the configuration managers 308 ofthe access points 106 of such changes. Thus, as service providers modifyconfiguration of their services and global network addresses at a datacenter 110, such modifications may be propagated to the access points106. In one embodiment, access points 106 do not store configurationinformation in persistent storage (and may lack any such persistentstorage) in order to reduce a likelihood that such information might beobtained from the access points 106.

While examples are discussed above with respect to a network-accessibleservice, each access point server 402 may be associated with multipleservices. For example, where each flow manager 304 is tasked withhandling packets addressed to a subset of global network addresseswithin a group of such addresses, each flow manager 304 may as aconsequence handle traffic addressed to any service associated with aglobal network address within that subset.

While only some components of the access points 106 are shown as incommunication with the networks 104 and 108, other components mayadditionally be in communication with the network 108 and/or the network104. The lines of FIG. 3 are not intended to represent all actual orpotential network connections, but rather to illustrate a possible flowof service-related traffic through an access point 106.

The data center 110 of FIG. 2 and the global access point 106 of FIG. 3are as operating in a distributed computing environment including one ormore computer systems that are interconnected using one or more computernetworks (not in the respective figures). The data center 110 of FIG. 2and the global access point 106 of FIG. 3 could also operate within acomputing environment having a fewer or greater number of devices thanare illustrated in the respective figures. Thus, the depictions of thedata center 110 of FIG. 2 and the global access point 106 of FIG. 3should be taken as illustrative and not limiting to the presentdisclosure. For example, data center 110 of FIG. 2, the global accesspoint 106 of FIG. 3, or various constituents thereof, could implementvarious Web services components, hosted or “cloud” computingenvironments, and/or peer to peer network configurations to implement atleast a portion of the processes described herein.

FIG. 4 depicts a general architecture of an example computing system(referenced as an access point server 402) that operates to implementflow managers 304 of an access point 106. The general architecture ofthe access point server 402 depicted in FIG. 4 includes an arrangementof computer hardware and software modules that may be used to implementaspects of the present disclosure. The hardware modules may beimplemented with physical electronic devices, as discussed in greaterdetail below. The access point server 402 may include many more (orfewer) elements than those shown in FIG. 4. It is not necessary,however, that all of these generally conventional elements be shown inorder to provide an enabling disclosure. Additionally, the generalarchitecture illustrated in FIG. 4 may be used to implement one or moreof the other components illustrated in FIGS. 2 and 3. As illustrated,the access point server 402 includes one or more processing units 490,one or more network interface 492, and one or more computer readablemedium drives 494, all of which may communicate with one another by wayof a communication bus. The network interfaces 492 may provideconnectivity to one or more networks or computing systems, such as therouter 302 (which may correspond, for example, to a commerciallyavailable router device). The processing units 490 may thus receiveinformation and instructions from other computing systems or servicesvia networks, such as network 104 or 108. The processing unit 490 mayalso communicate to and from memory 480.

The memory 480 may contain computer program instructions (grouped asmodules in some embodiments) that the processing unit 490 executes inorder to implement one or more aspects of the present disclosure. Thememory 480 generally includes random access memory (RAM), read onlymemory (ROM) and/or other persistent, auxiliary or non-transitorycomputer readable media. The memory 480 may store an operating system482 that provides computer program instructions for use by theprocessing unit 490 in the general administration and operation of theaccess point server 402. The memory 480 may further include computerprogram instructions and other information for implementing aspects ofthe present disclosure. For example, in one embodiment, the memory 480includes one or more flow manager units 483, each of which representscode executing to implement a flow manager 304 of FIG. 3. Each flowmanager unit 483 may illustratively be isolated from other units 483 onthe server 402. For example, each unit may represent a separate virtualmachine or isolated software container. In some instances, each unit 483may be associated with separate processing units 490, interfaces 492, ordrives 494, minimizing potential for operation of one unit 483 to impactoperation of another unit 483. Each unit 483 illustratively includes anendpoint selector unit 484, representing code executable to select anendpoint 202 to which to route a packet addressed to a global networkaddress, a flow table 486 representing a table of information mappingflows of packets to endpoints 202, a NAT unit 488 representing codeexecutable to perform NAT on packets addressed to global networkaddresses, or responses to such packets from endpoints 202, and asession handoff unit 489 representing code executable to conduct aninitialization phase of a connection-oriented communication session, andto handoff that session to a receiving device. While not shown in FIG.4, the memory 480 also illustratively includes an encapsulation unitrepresenting code executable to generate a tunnel connection to anotherdevice enabling transmission of encapsulated packets, and to conductencapsulation/decapsulation to facilitate such transmission.

The memory 480 may further include a health check unit 496,corresponding to instructions executable to implement the health checkdevice of 306, and a configuration manager unit 498, corresponding toinstructions executable to implement the configuration manager 308. Insome embodiments, the health check device of 306 and configurationmanager 308 may be implemented as standalone devices, rather than aspart of the access point server 402. Moreover, while shown as distinctfrom the access point server 402, the router 302 may in some instancesbe incorporated into the server 402 (e.g., by including in memory 480software executable to implement routing functionality).

In one embodiment, the access point server 402 lacks any non-volatilememory, and is configured to operate solely with respect to volatilememory, such as RAM. Illustratively, the access point server 402 may beconfigured to use a pre-boot execution environment (PXE) such that, oninitialization, the access point server 402 retrieves from anetwork-accessible storage location (e.g., the configuration data store210) at least a portion of the contents of the memory 480, such as theoperating system 482, health check unit 496, and configuration managerunit 498. The configuration manager unit 498 may thereafter retrieveadditional configuration information from a network-accessible storagelocation, such as configuration for individual services and associatedglobal network addresses, and utilize such additional information togenerate flow manager units 483. To prevent unauthorized disclosure ofthe contents of memory 480, authentication of the server 402 at thestorage location may be linked at least partly to a network location ofthe server 402 (e.g., at an access point 106), such that attempts tophysically relocate the server 402 result in a failure to retrieve thecontents of memory 408.

While FIG. 4 depicts a single server 402 and router 302, in someinstances a global access point 106 may be implemented by multipleservers 402 and/or routers 302. In some instances, such servers 402 orrouters 302 may be physically or logically isolated, to avoidpropagation of errors between such servers 402 or routers 302.Illustratively, where an access point 106 handles multiple pools ofnetwork addresses, each pool may be handled by a distinct server 402 androuter 302. Thus, should one router 302 and/or server 402 fail, onlyservices associated with the pool handled by that router 302 and/orserver 402 would be expected to be affected.

With reference to FIG. 5, illustrative interactions will be describedthat depict how an individual global access point 106A may operateresiliently, by providing multiple routes to reach a network-accessibleservice utilizing global network addresses. FIG. 5 depicts anenvironment 500, which may in turn represent an embodiment of a portionof the environment 100 of FIG. 1. Specifically, in the environment 500,the network 104 is divided into networks 104A-E, each of which mayrepresent, for example, an autonomous system. The networks 104A-C areillustrated as in communication with client devices 102. These network104A-C may represent, for example, Internet service providers (ISPs) ofclient devices 102. Networks 104D and E represent other AS's to whichthe global access point 106A has a network connection. While networks104D and E are not shown in FIG. 5 as connected to client devices 102,such networks may also act as ISPs to client devices 102. Each of thenetworks 104A-E is shown in FIG. 5 as interconnected to other networks104A-E. This configuration is illustrative, and may vary acrossimplementations.

In the illustration of FIG. 5, client devices 102 generally have tworoutes through which to reach the global access point 106A: via network104D and 104E. To increase the resiliency of access to the global accesspoint 106A, the access point 106A may selectively transmit availabilityannouncements to the networks 104D and 104E. Specifically, as shown inFIG. 5, at (1), a global access point 106A may identify at least twogroups of global network addresses, such as two contiguous range ofnetwork addresses. Thereafter, rather than advertising all globalnetwork addresses to both network 104D and E, the global access point106A, at (2), transmits an advertisement (e.g., a BGP announcement) forthe first network group to the network 104D. Similarly, at (3), theglobal access point 106A transmits an advertisement (e.g., a BGPannouncement) for the second network group to the network 104E. In thisway, traffic addressed to addresses within the first network group arelikely to arrive at the access point 106A through network 104D, andtraffic addressed to addresses within the second network group arelikely to arrive at the access point 106A through network 104E. Eachservice may be associated with a global network addresses of at leasttwo groups. Thus, if errors or issues occur on the either of thenetworks 104D or E, a client device 102 may utilize an alternativeglobal network address of a service in order to access the global accesspoint 106A through the remaining network.

With reference to FIG. 6, illustrative interactions will be describedfor operation of a global access point 106 to facilitate communicationbetween a client device 102 and a data center 110 providing endpoints202 for a network-accessible service. The interactions of FIG. 6 beginat (1), where a client device 102 transmits a data packet to the globalaccess point 106A, addressed to a global network address associated withthe global access point 106A. The data packet may, for example, beformatted in accordance with the Transmission Control Protocol (TCP),the user datagram protocol (UDP), or the Internet Control MessageProtocol (ICMP). The data packet may be transmitted via the network 104,and routed to the global access point 106A via operation of the network104, based on advertisements of the global access point 106A indicatingthat the global network address to which the packet is addressed isavailable via the global access point 106A. For example, the globalaccess point 106A may be a nearest (e.g., in network distance terms)global access point 106 to the client device 102.

On receiving the data packet, a router 302 within the global accesspoint 106A assigns the packet to a flow manager 304 within the accesspoint 106. Where the data packet is not associated with an existing dataflow, the router 302 may utilize ECMP load balancing to assign the datapacket to a flow manager 304 based on the network address to which thepacket is addressed. Where the data packet is associated with anexisting data flow, the router 302 may route the packet to the same flowmanager 304 to which prior packets within the flow were routed.

At (2), the flow manager 304 within the access point 106A assigns thedata packet to a data center 110. Where the data packet is notassociated with an existing data flow, the flow manager 304 may apply aload balancing algorithm to the data packet, to select a data center 110from the set of available data centers 110. The algorithm may include,for example, initially selecting a data center 110 that is mostperformant with respect to the client device 102 (or, alternatively, theglobal access point 106A) in terms of network performance criteria, andselecting an alternative data center 110 (e.g., a next most performant)if the nearest data center 110 has received greater than a thresholdamount (e.g., in absolute amount or percentage) or data packets over apast period of time. In one embodiment, each access point 106Aimplements the same algorithm when selecting a data center 110, based oncurrent information related to performance criteria between clientdevices 102 and data centers 110. Moreover, a common set of performancecriteria may be made available to each access point 106 (e.g., fromwithin configuration data store 210). Thus, absent a change inperformance criteria or load of data centers 110, routing of clientdevice 102 data packets to a data center 110 may be consistent amongaccess points 106. In this manner, even when data packets of a clientdevice 102 are rerouted to a different access point 106, they may berouted to the same data center 110. Where the data packet is associatedwith an existing data flow, the flow manager 304 may route the packet tothe same data center 110 to which prior packets within the flow wererouted.

After selecting a data center 110, the flow manager 304 may select anendpoint 202 within the selected data center 120. The flow manager 304may illustratively maintain, for a given data center 110, a set ofendpoints 202 providing a network accessible service associated with theglobal network address to which the data packet was addressed. In oneembodiment, this set of endpoints 202 may be identified based oninformation received from a resource manager 206 within the data center110. The flow manager 304 may further maintain information as to theapparent health of the endpoints 202, as obtained from a health checkdevice 204 within the data center 110, a health check device 306 withinthe access point 106A, or both. The flow manager 304 may thus selectfrom a set of endpoints 202 a healthy endpoint for the serviceassociated with the network address to which a packet was addressed. Theflow manager 304 may utilize any of a number of known load balancingalgorithms, such as consistent hashing, to distribute packets betweenendpoints 202. Where a packet is associated with an existing packetflow, the flow manager 304 may select the same endpoint 202 to whichprior packets in the flow have been routed.

After selecting an endpoint, the flow manager 304 within the accesspoint 106A modifies the packet as necessary to route the packet to theselected endpoint 202. Specifically, in the embodiment of FIG. 6, theflow manager 304 at (5) applies NAT techniques to the packet, such as byreplacing source and destination network addresses of the packet.Thereafter, at (6), the flow manager 304 transmits the packet toselected endpoint 202 within the selected data center 110.

At (7), the endpoint 202 processes the packet in accordance with thenetwork-accessible service. The endpoint 202 then returns a responsepacket to the global access point 106A. As the response packet isassociated with an existing data flow, the router 302 within the accesspoint 106A directs the response packet the same flow manager 304discussed above. The flow manager 304 reverses the previous NATtransformation, in order to address the packet to the client device 102.The flow manager 304 then returns the data packet to the client 102.Thus, via the interactions of FIG. 6, a client device 102 may address adata packet to a global network address, and be routed (via a globalaccess point 106) to an endpoint providing a network-accessible servicerelated to the network address. The access point 106 may applyload-balancing to packets, such that load on the service is distributedamong data centers 110, in accordance with a desired configuration of aservice provider.

With reference to FIG. 7, illustrative interactions will be describedfor propagating endpoint information from a data center 110A to globalaccess points 106, such that the access points 106 may correctly routetraffic from client devices 102 addressed to global network addresses.The interactions of FIG. 7 will be described with respect to a singledata center 110A. However, similar interactions may occur with respectto other data centers 110, potentially concurrently.

The interactions of FIG. 7 begin at (1), where the resource manager 206of a data center 110A detects an endpoint scaling event within theendpoint pool 201. An endpoint scaling event illustratively correspondsto a “scale up” or “scale down” event, indicating an addition or removalof an endpoint 202 from the pool 201. In some instances, a scaling eventmay also include replacement of an endpoint 202 in the pool 201. Scalingevents may illustratively occur due to load on the pool 201 fallingoutside of desired threshold values, or due to malfunction of anendpoint 202 within the pool 201. At (2), the resource manager 206transmits instructions to the pool 201 to alter the pools configuration,in accordance with the scaling event. For example, the resource manager206 may instruct the pool 201 to add a new endpoint 202, remove anexisting endpoint 202, or a combination thereof. Illustratively, scalingthe pool 201 may include transmitting instructions to a host device toadd or remove a virtual machine instance executing softwarecorresponding to the endpoint 202 (e.g., making available anetwork-accessible service related to the pool 201).

At (3), the pool 201 notifies the resource manager 206 of modificationsto the pool 201, including for example network addresses of anyendpoints 202 added to the pool 201. The resource manager 206, at (4),notifies the configuration manager 208 of any network addressmodifications to the pool 201. A variety of techniques may be utilizedin accordance with the present disclosure to facilitate transmission ofinformation between the resource manager 206 and the configurationmanager 208. Preferably, such communication will occur rapidly (e.g.,within milliseconds or seconds), such that future requests to access anetwork-accessible service are routed to an available endpoint 202, asopposed to a failed or unavailable endpoint 202. In one embodiment, thisrapid communication is facilitated by a pub/sub messaging system, astream-data processing system, or a combination thereof. Illustratively,the resource manager 206 may publish modifications to the endpoints 202of a pool 201 within a message stream associated with thenetwork-accessible service provided by the pool 201. The configurationmanager 208 may be configured to retrieve new messages to the stream,either by periodically polling for such messages (e.g., at a frequencyof every 1 second, 2 seconds, 3 seconds, etc.) or by receiving “push”notifications for such messages by the message-handling system. Use of apub/sub messaging system or stream-data processing system maybeneficially provide highly resilient, rapid messaging between theresource manager 206 and the configuration manager 208. While describedwith reference to changes to network addresses, a pub/sub messagingsystem or stream-data processing system may be used to facilitatecommunication regarding other changes to configuration of global accesspoints 106.

On detecting a change to the network addresses of the pool 201, theconfiguration manager 208 may modify a configuration of the globalaccess points 106 to reflect those changes, as detailed below. While theinteractions below are described with reference to alterations ofnetwork addresses within a pool 201, the same or similar interactionsmay be utilized to propagate other configuration changes to accesspoints 106.

Specifically, at (5), after being determining a change in networkaddress of endpoints 202 within the pool 201, the configuration manager208 builds a modified configuration package (or parcel within a package)for the global access points 106. The modified configuration package (orparcel) may illustratively reflect the changes to the network addressesof the pool 201, as well as any other changes to configuration of theglobal access points 106 since a last build of the configuration package(or parcel). In one embodiment, the configuration package (or parcel)reflects only a change since a last version of the package (or parcel).In another embodiment, the package (or parcel) reflects all relevantconfiguration information in the package (or parcel). At (6), theconfiguration manager 208 stores the modified configuration package (orparcel) in the data store 210 for retrieval by the global access points106.

As noted above, in one embodiment, each global access point 106 isconfigured to periodically poll the configuration data store 210 forchanges to its configuration package (or parcels thereof). Thus, at (7),each global access point 106 can determine that a modified package (orparcel) exists in the data store 210, and retrieve that package (orparcel). Each access point 106 can then, at (8), update itsconfiguration (e.g., with the changes to network addresses within thepool 201), such that future requests are routed to a correct endpoint202 within the pool 201. The illustrative interactions of FIG. 7 canthus facilitate rapid modification to the configuration of global accesspoints 106, such as in response to modification of endpoint pools 201.

With reference to FIG. 8, an illustrative routine 800 will be describedfor increasing resiliency of global network addresses, by selectingadvertisement of addresses to different neighboring devices. The routine800 may illustratively be implemented by a global access point 106(e.g., during initialization of the access point 106). In oneembodiment, an instance of the routine 800 is implemented by each accesspoint 106.

The routine 800 begins at block 804, where global network addresses tobe serviced via the access point 106 are obtained. This information mayillustratively be obtained as part of a configuration of the accesspoint 106, such as within a configuration file stored in a configurationdata store 210, obtained during initialization of the access point 106.The global network addresses may be illustratively divided in theconfiguration into different addressing pools. For example, each poolmay include a different “block” of network addresses, such as acontiguous range of addresses. Ranges may represent “subnets” forexample. In one embodiment, ranges are represented by a “prefix,”denoting the first n bits of network addresses in the range. Forexample, the prefix of 192.0.2.0/24 (expressed in Classless Inter-DomainRouting, or CIDR) may represent the first 24 bits of an IPv4 address,corresponding to addresses in the range of 192.0.2.0 to 192.0.2.255.

At block 808, the access point 106 selects neighboring devices to whichto advertise each pool of network addresses. Generally, the availablepools may be divided among available neighboring devices, in order toprovide different network paths from which to reach the access point. Inone embodiment, such election is based on a specification within theconfiguration of the access point 106. In another embodiment, suchselection is determined by the access point 106. Illustratively, theaccess point 106 may divide the pools evenly among available neighboringdevices (which may be discovered, e.g., through typical routingprotocols). In some instances, neighboring devices may additionally bedetermined based on preferred routing of traffic. For example, an accesspoint 106 may be configured not to advertise a given network addresspool on a given neighboring device, in order to cause that neighboringdevice to route requests for addresses within the pool to an alternativeaccess point 106.

Thereafter, at block 810, the global access point 106 transmits to therelevant neighboring devices availability announcements for therespective pools of network addresses. The announcements may, forexample, be BGP protocol announcements. Transmission of BGP protocolannouncements generally is known within the art, and thus will not bedescribed in detail herein.

As described above, the division of global network address into pools,and advertisement of such pools to different neighboring devices, maybeneficially increase the resiliency of access points 106, particularlyto outages or errors on neighboring networks. Specifically, because eachnetwork-accessible service may be associated with global networkaddresses of different pools (e.g., one for each pool), client devices102 of a service may be made aware (e.g., via DNS) of multiple routesfor reaching the service. This technique may further increase theresiliency of access points 106 to limited outages at the access points106, such as outages of a network interface or router connected to aneighboring network. As noted above, in one embodiment, the resiliencyof each service is further increased by dividing each pool into networkaddress subsets, which may be in turn distributed among flow managers304 of an access point 106. The associations between services and eachsubset may be “shuffled” among pools. For example, the global networkaddress for a service in a given pool may be randomly selected, or maybe selected via a varied selection mechanism for each pool. This maycause the groupings of services within subsets to be “shuffled” amongthe pools. In this manner, if an individual service's configuration isproblematic to other services within the subset, the other servicesaffected by that misconfiguration are likely to vary among pools.Because each service may be expected to be accessible via addresses inmultiple pools, client devices 102 may connect to each other service viaan alternative address of an alternative pool, thereby bypassing theproblematic subset and pool. This shuffling mechanism can thereforegreatly increase resiliency of services using global network addresses.

With reference to FIG. 9, an illustrative routine 900 will be describedfor routing traffic addressed to a global network address associatedwith a service provided by endpoints 202 within data centers 110. Theroutine 900 may illustratively be implemented by a flow manager 304within an access point 106. In one embodiment, an instance of theroutine 900 is implemented by each flow manager 304 of each access point106.

The routine 900 begins at block 902, wherein the flow manager 304receives a data packet addressed to a global network addressesadvertised as accessible at the access point 106. The data packet may,for example, represent a request to access a network-accessible servicemade available via the network address. In one example, the data packetis formatted as a TCP or UDP data packet.

At block 903, the routine 900 varies according to whether the datapacket is associated with an existing packet flow, as determined atblock 903. Illustratively, the access point 106 may compare attributesof the data packet to attributes of prior packets, to determine whetherthe new data packet is within a same flow of communication as a priorpacket. Any number of flow identification techniques, a variety of whichare known in the art, may be used to determine whether a data packet iswithin an existing flow.

If the packet is part of an existing flow, the routine 900 proceeds toblock 913, where the flow manager 304 selects as an endpoint 202 for thepacket the same endpoint 202 selected for previous packets within theflow. In one embodiment, each flow manager 304 may maintain a cache inmemory associating flows to endpoints 202, facilitating this selection.

If the packet is not part of an existing flow, the routine 900 proceedsto block 904, where the flow manager 304 identifies thenetwork-accessible service associated with the network address to whichthe packet is addressed. The flow manager 304 may illustratively accessinformation mapping network addresses to associated services (e.g., asmaintained in memory of the access point 106) to identify the serviceassociated with the address.

Thereafter, at block 906, the access point 106 selects a data center 110providing the network-accessible service. In one embodiment, the datacenter 110 is selected at least partly based on network performancecriteria between the access point 106 and the data center 110, as wellas a load balancing configuration for the network-accessible service.Illustratively, the flow manager 304 may initially select a mostperformant (in terms of a given network performance criteria withrespect to the client device 102 or the access point 106) data center110. The flow manager 304 may then determine whether a proportion ofpackets routed to the data center 110 over a past period (e.g., 1minute, 5 minutes, 30 minutes, etc.) exceeds a threshold desired maximumfor the data center 110, based on a desired load balancing across datacenters 110 for the service. In one embodiment, the threshold desiredmaximum may be specified, for example, by a service provider, such asvia a desired percentage of traffic to be routed to each data center110. In another embodiment, the threshold desired maximum may bedetermined cooperatively by multiple access points 106, such that thecombined thresholds of each access point 106 achieve the desireddivision of a service provider between data centers 110. For example,access points 106 may implement a peer-to-peer algorithm that aggregatestraffic volumes to a service across the access points 106 and determinesan optimal routing for each data packet based on the access point 106 tothe packet was addressed (and corresponding network performance criteriato each data center 110). The algorithm may then modify the optimalrouting for each access point 106 as necessary to implement globallyoptimized routing, resulting in an individualized desired proportionbetween data centers 110 for each access point 106. Each access point106 may thus implement this individualized desired proportion todetermine whether a proportion of packets routed to the initiallyselected data center 110 over a past period (e.g., 1 minute, 5 minutes,30 minutes, etc.) exceeds a threshold desired maximum for the datacenter 110. If so, the access point may modify the initial selection toan alternative data center, such as a next closest data center. In oneembodiment, implementation of block 906 repeats for each data center 110selected, to ensure that no data center receives greater than a desiredmaximum proportion or volume of data packets.

After selecting a data center 110, the flow manager 304 selects anendpoint 202 within the data center 110 to which to route the datapacket. In one embodiment, the flow manager 304 may utilize consistenthashing to select an endpoint 202, based on an attribute of the packet(e.g., source IP address).

Thereafter, at block 910, the flow manager 304 applies NAT to the datapacket. Illustratively, the flow manager 304 may replace a sourceaddress of the data packet with an address of the global access point106 (e.g., a unicast address, as opposed to a global network address),and replace a destination address with an address of the endpoint 202(which may also be a unicast address of the endpoint 202). The flowmanager 304 may also, for example, modify a port number of the packet,to facilitate later application of NAT techniques to packets within theflow. The flow manager 304 may then transmit the packet to the selectedendpoint 202. The routine 900 may then end at block 914.

Various modifications to the routine 900 are contemplated herein. Forexample, while the routine 900 includes three blocks related toselection of an endpoint 202 for packets not associated with an existingflow (blocks 904, 906, or 908), some implementations of the routine 900may exclude one or more of these blocks. Illustratively, rather thanseparately identifying a service and data centers related to thatservice, a flow manager 304 may maintain data mapping a network addressto data centers 110 and endpoints 202 associated with the address. Theflow manager 304 may then select a data center 110 based on the criteriadescribed above, without the need to directly identify the serviceassociated with the address. In other embodiments, the flow manager 304may maintain only data associating network addresses to correspondingendpoints 202, and may select an endpoint 202 for a packet directly,without first identifying the service associated with the address or adata center 110 providing the service. For example, each endpoint 202may be selected in a manner similarly to the data centers 110 describedabove, by selecting an endpoint 202 with a best performance criteriavalue, and modifying that selection if the endpoint 202 is receivingmore than a desired proportion of volume of data. Moreover, while theroutine 900 is shown in FIG. 9 as ending at block 914, a flow manager304 may continue to undertake other actions subsequent to the routine900, such as acting as a NAT device for communications between clientdevices 102 and endpoints 202. Thus, the depiction of the routine 900 inFIG. 9 is one embodiment of the concepts disclosed herein.

With reference to FIG. 10, an illustrative routine 1000 will bedescribed for updating information at the global access points 106regarding endpoints 202 of a data center 110 that provide anetwork-accessible service. The routine 1000 may illustratively beimplemented by a resource manager 206 within a data center 110. In oneembodiment, an instance of the routine 900 is implemented by eachresource manager 206 of each data center.

The routine 1000 begins at block 1002, where the resource manager 206 ofa data center 110A detects an endpoint scaling event within an endpointpool 201. An endpoint scaling event illustratively corresponds to a“scale up” or “scale down” event, indicating an addition or removal ofan endpoint 202 from the pool 201. In some instances, a scaling eventmay also include replacement of an endpoint 202 in the pool 201. Scalingevents may illustratively occur due to load on the pool 201 fallingoutside of desired threshold values, or due to malfunction of anendpoint 202 within the pool 201.

At block 1004, the resource manager 206 alters the pool configurationbased on the detected scaling event. Illustratively, the resourcemanager 206 may transmit instructions to the pool 201 to alter the poolsconfiguration, in accordance with the scaling event. For example, theresource manager 206 may instruct the pool 201 to add a new endpoint202, remove an existing endpoint 202, or a combination thereof.Illustratively, scaling the pool 201 may include transmittinginstructions to a host device to add or remove a virtual machineinstance executing software corresponding to the endpoint 202 (e.g.,making available a network-accessible service related to the pool 201).

At block 1006, the resource manager 206 obtains from the pool 301network addresses of any endpoints 202 added to or removed from the pool201. Further, at block 1008, the resource manager 206 notifies theglobal access points 106 of any network address modifications to thepool 201. As discussed above, the notifications may in some instances betransmitted via alterations to a configuration package (or parcel) forthe access points 106. Thus, the global access points 106 may updatetheir endpoint availability for the network-accessible service relatedto the pool 201, such that future requests to access the networkaccessible service can be routed to an available endpoint 202 in thepool 201. The routine 1000 can then end at block 1010.

With reference to FIG. 11, illustrative interactions will be describedfor conducting an initialization phase of a connection-orientedcommunication session at a global access point 106A, and handing off thesession to a device within a data center 110, thus reducing the timerequired to establish such a session. The interactions of FIG. 11 may beimplemented, for example, by a flow manager 304 of an access point 106.The interactions of FIG. 11 will be described with respect to a singleaccess point 106A and data center 110A. However, similar interactionsmay occur with respect to other access points 106 and data centers 110,potentially concurrently. Moreover, the interactions of FIG. 11 will bedescribed with respect to a specific type of connection-orientedcommunication session: a TCP session. Similar interactions may occurwith respect to other connection-oriented communication sessions.

The interactions of FIG. 11 begin at (1), where a client device 102attempts to initiate a connection-oriented communication session with anetwork service, by transmitting a TCP SYN packet addressed to a globalnetwork address of the service. In accordance with the functionalitiesdescribed above, the TCP SYN packet is transmitted to the global accesspoint 106A, as shown in FIG. 11. The TCP SYN packet illustrativelyincludes a sequence number for the client device 102.

At (2), a flow manager 304 of the global access point 106A continues theTCP three-way handshake by returning to the client device 102 a TCPSYN-ACK packet, which illustratively includes a sequence number for theaccess point 106A (as well as acknowledging the client device's 102sequence number). At (3), the client device 102 continues the TCPthree-way handshake by returning to the access point 106A a TCP ACKpacket, acknowledging the sequence number of the access point 106A. Onreception of the TCP ACK packet, a TCP session is initiated between theclient device 102 and the global access point 106A. Because the accesspoint 106A is expected to be nearby to the client device 102 (e.g., interms of latency), interactions (1)-(3) are expected to completequickly, relative to conducting the initialization phase between theclient device 102 and an endpoint 202.

At (4), the client device 102 (understanding the TCP session to havebeen initialized with the service), transmits a data packet within theTCP session, addressed to the global network address of the service. Thedata packet is routed to the global access point 106A, which at (5),selects a data center 110 to which to route the packet. Selection of adata center may occur in the same manner as described above (e.g., asinteraction (3) of FIG. 6). The access point 106 further, at (6)encapsulates the data packet for transmission to the selected datacenter 110 (in this instance, data center 110A) via a tunnel. In theinteractions of FIG. 11, it is assumed that the access point 106A haspreviously established a tunnel to the data center 110A (e.g., to asession handoff manager 212 of the data center 110A). For example, theaccess point 106A may maintain one or more idle TCP tunnels to the datacenter 110A for use in transmitted packets to the data center 110.However, an additional interaction may be included, wherein the globalaccess point 106A establishes a tunnel to the data center 110A (e.g., aUDP or TCP tunnel). To facilitate handoff of the established TCPsession, the encapsulated data packet also illustratively includes TCPsession state information, such as the five-tuple for the TCP sessionand sequence numbers of the session. In the embodiment illustrativelydepicted in FIG. 11, the session state information is included as headerinformation for the encapsulated data packet. At (7), the global accesspoint 106A transmits the encapsulated packet to the data center 110A(e.g., to a session handoff manager 212 within the data center 110A).

At (8), on receiving the encapsulated packet, the device within the datacenter 110A (e.g., the session handoff manager 212, or in some instancesan endpoint 202) adopts the TCP session, by adding information indicatedwithin the encapsulated packet (e.g., within a header) to its own TCPsession state table. The device then, at (9), decapsulates the packet ofthe client device 102, and at (10) processes it as if it were receiveddirectly at the device. For example, where the device is an endpoint202, the endpoint 202 may process the data of the packet in accordancewith the service requested by the client device 102. Where the device isa session handoff manager 212, the manager 212 may process the packet byidentifying an endpoint 202 to service the packet (e.g., in a mannersimilar to interaction (4) of FIG. 6), and utilize the endpoint 202 toservice the packet. Illustratively, because the session handoff manager212 (and in this case not the endpoint 202) is a party to the TCPconnection with the client device 102, the manager 212 may initiate asecond TCP connection with the endpoint 202, and pass the data withinthe client devices 102 packet to the endpoint 202 via the second TCPsession. The manager 212 may continue to operate as a proxy between theclient device 102 and the endpoint 202 for future communications. Forexample, the manager 212 may obtain a response from the endpoint 202 andfacilitate transmission of the response to the client device 102 via theTCP session with the client device 102.

At (11), the device within the data center 110A transmits a responsepacket to the global access point 106A, such as via the tunnel.Interaction (11) may include, for example, encapsulating the responsepacket for transmission via the tunnel. The global access point 106Athen, at (12), forwards the response packet to the client device 102.Interaction (12) may include, for example, decapsulating the responsepacket for transmission to the client device 102. Further communicationsbetween the client device 102 and an endpoint 202 within the data center110A may occur in a similar manner to interactions (4) through (7) and(9) through (12), above. Thus, the client device 102 may communicatewith the data center 110A via a TCP session, without actually beingrequired to complete an initialization phase of the session with adevice of the data center 110A.

While FIG. 11 depicts illustrative interactions, these interactions mayvary across embodiments. For example, while a response packet isdepicted in FIG. 11 as passing through the global access point 106A, insome instances an endpoint 202 may be configured to respond directly toa client device 102, without the response passing through the accesspoint 106A. For example, rather than encapsulating a response packet andtransmitting it to the access point 106A, an endpoint 202 (or manager212) may transmit the response packet directly to the client device 102(e.g., over network 104), avoiding the need for encapsulation. Asanother variation, while session information is described in FIG. 11 asincluded within an encapsulated data packet, in other embodiments,session state information may be included in a separate packet. Forexample, a global access point 106A may be configured to separatelytransmit session state information to a data center 110A afterconducting an initialization phase of a communication session, such asin a “session adoption” command to the data center 110A. As yet anothervariation, while the manager 212 is discussed above as selecting anendpoint 202 within the data center 110A to service a client device 102,in some instances an access point 106 may select such an endpoint 202,even where a manager 212 is used. For example, an access point 106 mayselect an endpoint 202, and notify the manager 212 of which endpoint hasbeen selected (e.g., in headers of an encapsulated packet). In instanceswhere the global access point 106 selects an endpoint 202 to which toroute a packet, the access point 106 may modify the packet to facilitaterouting to the endpoint 202. For example, the access point 106 mayreplace, within a destination address filed of the packet, a globalnetwork address for the service with a network address of the endpoint202 (e.g., a “unicast” address). Thus, the interactions of FIG. 11 areintended to be illustrative in nature.

While the interactions of FIG. 11 depict interactions of a client device102 with a single global access point 106A, in some instances a clientdevice 102 may interact with multiple global access points 106. As notedabove, each global access point 106 may be configured to advertiseglobal network addresses to a public network (e.g., network 104), notunder control of the global access points 106. Thus, devices on such anetwork can generally determine the access point 106 to which packets ofa client device 102 addressed to a global network address are routed. Insome configurations, rerouting of packets within a given packet flow toa different global access point 106 can detrimentally affect clientdevice 102 communications. For example, where a client device 102establishes a TCP connection that requires state information to bemaintained at an individual access point 106 (e.g., a TCP connectionwith the access point 106 or NAT'ed through the access point 106),rerouting of client device 102 communications to a different accesspoint 106 can undesirably break that connection.

The interactions of FIG. 11 address this scenario, by enabling TCPconnections between a client device 102 and endpoint 202 (or sessionhandoff manager 212) to be maintained even when rerouting of clientdevice 102 packets to a different access point 106 occurs. Specifically,each access point 106 can be configured to apply the same load balancingcriteria when selecting a data center 110 or endpoint 202 to which toroute a client device 102's packet. Such load balancing criteria may beaccess point 106 agnostic (e.g., invariant regardless of the accesspoint 106 at which it is applied). For example, the load balancingcriteria may reflect latency between a client device 102 and the datacenter 110, as well as health of the data center 110 (or endpoints 202therein), independent of latency to or from the access point 106. Assuch, each access point 106 can be expected to route a given clientdevice 102's packets to the same data center 110. Thus, for example, ifthe client device 102 transmitted an additional packet to a secondaccess point 106, the second access point 106 would apply the same loadbalancing criteria to select the data center 110A as a destination forthe packet. The second access point 106 would then route the packet tothe endpoint 202 (or session handoff manager 212), which would processthe packet in the same way as if it had been routed through the accesspoint 106A. Because the interactions of FIG. 11 do not require stateinformation to be maintained at the access point 106A, and because theencapsulation mechanism of FIG. 11 maintains a source network address ofthe client device 102 within encapsulated packets, no interruption of aconnection-oriented communication session (e.g., a TCP session) wouldoccur. Thus, the interactions of FIG. 11 address rerouting problems thatwould otherwise occur when utilizing anycast technologies to shaperouting of connection-oriented communication sessions.

With reference to FIG. 12, an illustrative routine 1200 will bedescribed for initialization of a connection-oriented communicationsession at a global access point 106A, and handoff of the session to amanager 212 within a data center 110. The routine 1200 is illustrativelyimplemented cooperatively between the global access point 106A and thesession to a manager 212, and thus the boxes of the routine 1200 aredivided among such devices. While portions of the routine 1200 isdepicted as being implemented by a manager 212, these portions mayalternatively be implemented directly within an endpoint 202 in someinstances.

The routine 1200 begins at block 1202, where the access point 106 (e.g.,a flow manager 304) obtains a request addressed to a global networkaddress of a service to initiate a connection-oriented communicationsession with the service. The request may be, for example, a TCP SYNpacket.

At block 1204, the access point 106 completes an initialization phase ofthe session, according to the particular protocol used for the session.Where the protocol is TCP for example, the initialization phase mayinclude conducting a three-way handshake with the client device 102.

At block 1206, the access point 106 receives a data packet from theclient device 102 within the session. For example, the packet may be apayload packet within a TCP session.

At block 1208, the access point 106 encapsulates the data packet, fortransmission to the manager 212 via a network tunnel. The access point106 further includes in the encapsulated packet (e.g., as a headerinformation for the encapsulated packet) session context for thecommunication session, such as a TCP five-tuple and sequence numbers.The access point 106 then, at block 1210, sends the packet to thehandoff manager 212 as a handoff request for the communication session.

At block 1212, the handoff manager 212 receives the encapsulated packet,and at block 1214, constructs within its local data a communicationsession, based on the context information from the access point 106. Themanager 212 thus adopts the session, enabling subsequent communicationswithin the session between the client device 102 and the manager 212. Atblock 1216, the handoff manager 212 decapsulates the data packet of theclient device 102, and processes the packet within the session. Forexample, the manager 212 may select and endpoint 202 to handle therequest, and transmit the content of the data packet to the endpoint 202via another communication session. The routine 1200 then ends at block1218. Thus, the client device 102 and the manager 212 may communicatevia the stateful session, with requiring the client device 102 and themanager 21 to communicate to establish the session.

The routine 1200 may include additional or alternative blocks than thosedescried above. For example, prior to sending an encapsulated packet asa handoff request at block 1210, the access point 106 may select a datacenter to receive the handoff request, in a manner similar to selectionof a data center discussed in FIG. 9. Moreover, while the routine 1200is depicted as ending subsequent to block 1216, the access point 106 andmanager 212 may continue to operate to facilitate communications of orto a client device 120 within the session, as discussed above. Thus, thenumber and arrangement of blocks in FIG. 12 is illustrative in nature.

All of the methods and processes described above may be embodied in, andfully automated via, software code modules executed by one or moregeneral purpose computers or processors. The code modules may be storedin any type of non-transitory computer-readable medium or other computerstorage device. Some or all of the methods may alternatively be embodiedin specialized computer hardware.

Conditional language such as, among others, “can,” “could,” “might” or“may,” unless specifically stated otherwise, are otherwise understoodwithin the context as used in general to present that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Disjunctive language such as the phrase “at least one of X, Y or Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y or Z, or any combination thereof (e.g., X, Y and/or Z).Thus, such disjunctive language is not generally intended to, and shouldnot, imply that certain embodiments require at least one of X, at leastone of Y or at least one of Z to each be present.

Unless otherwise explicitly stated, articles such as ‘a’ or ‘an’ shouldgenerally be interpreted to include one or more described items.Accordingly, phrases such as “a device configured to” are intended toinclude one or more recited devices. Such one or more recited devicescan also be collectively configured to carry out the stated recitations.For example, “a processor configured to carry out recitations A, B andC” can include a first processor configured to carry out recitation Aworking in conjunction with a second processor configured to carry outrecitations B and C.

Any routine descriptions, elements or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or elements in the routine. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, orexecuted out of order from that shown or discussed, includingsubstantially synchronously or in reverse order, depending on thefunctionality involved as would be understood by those skilled in theart.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure and protected by the following claims.

What is claimed is:
 1. A system comprising: a plurality of endpointpools, each endpoint pool being situated in a different geographiclocation and containing one or more endpoint computing devicesconfigured to provide a network-accessible service associated with anetwork address; and at least two access points to thenetwork-accessible service, each access point comprising a processor andconfigured with computer-executable instructions to: periodicallyretrieve configuration information for the at least two access pointsfrom a network-storage location accessible to the at least two accesspoints, the configuration information including at least addressinginformation for each endpoint pool of the plurality of endpoint pools,the addressing information each endpoint pool including addresses ofindividual endpoint computing devices within the endpoint pool, whereinthe configuration information includes information regarding multiplesubsets of network-addresses accessible via each access point, andwherein the network address is included within a first subset of themultiple subsets; implement a plurality of isolated flow managers,different flow managers configured to route packets associated withdifferent subsets of network addresses from the multiple subsets; at afirst flow manager of the plurality of isolated flow managers that isconfigured to route packets associated with the first subset receive afirst network packet addressed to the network address; select a firstendpoint pool, from the plurality of endpoint pools, to which to routethe first network packet; and based at least partly on the addressinginformation for the first endpoint pool, route the first network packetto an individual endpoint computing device within the first endpointpool; receive modified configuration information for the access point,the modified configuration information reflecting modified addressinginformation for the first endpoint pool as a result of a change in theone or more endpoint computing devices included within the firstendpoint pool; and at the first flow manager: receive a second networkpacket addressed to the network address; select the first endpoint pool,from the plurality of endpoint pools, as a destination for the secondnetwork packet; and based at least partly on the modified addressinginformation for the first endpoint pool, route the second network packetto an individual endpoint computing device within the first endpointpool.
 2. The system of claim 1, wherein each access point providesaccess to a plurality of network-accessible services, and wherein theconfiguration information includes a mapping of each network-accessibleservice and addressing information for each endpoint pool of a pluralityof endpoint pools for the network-accessible service.
 3. The system ofclaim 1 further comprising a configuration manager configured to:receive information indicating the change in the one or more endpointcomputing devices included within the first endpoint pool; and generatethe modified configuration information, wherein the modifiedconfiguration information reflects a relative difference in theconfiguration information as compared to a prior version of theconfiguration information.
 4. The system of claim 3, wherein theinformation indicating the change in the one or more endpoint computingdevices included within the first endpoint pool corresponds to anotification that an unhealthy endpoint computing device will be removedfrom the first endpoint pool.
 5. The system of claim 1, wherein eachaccess point lacks non-volatile memory in which to store thecomputer-executable instructions, and wherein each access point isfurther configured, on initialization, to retrieve thecomputer-executable instructions from a network-accessible storagelocation and store the computer-executable instructions in volatilememory of the access point.
 6. A method implemented at a plurality ofgeographically-distributed access points to a network-accessible serviceassociated with a network address, the method comprising, at each accesspoint: periodically retrieving configuration information from anetwork-storage location accessible to the plurality of access points,the configuration information common to all access points and includingat least addressing information for each endpoint pool of a plurality ofendpoint pools, the addressing information for each endpoint poolincluding addresses of individual endpoint computing devices within theendpoint pool configured to provide the network-accessible service,wherein the configuration information includes information regardingmultiple subsets of network-addresses accessible via each access point,and wherein the network address is included within a first subset of themultiple subsets; implementing a plurality of isolated flow managers,different flow managers configured to route packets associated withdifferent subsets of network addresses from the multiple subsets; at afirst flow manager of the plurality of isolated flow managers that isconfigured to route packets associated with the first subset; receivinga first network packet addressed to the network address; selecting afirst endpoint pool, from the plurality of endpoint pools, to which toroute the first network packet; and based at least partly on theaddressing information for the first endpoint pool, routing the firstnetwork packet to an individual endpoint computing device within thefirst endpoint pool; receiving modified configuration information forthe access point, the modified configuration information reflectingmodified addressing information for the first endpoint pool as a resultof a change in the one or more endpoint computing devices includedwithin the first endpoint pool; and at the first flow manager: receivinga second network packet addressed to the network address; selecting thefirst endpoint pool, from the plurality of endpoint pools, as adestination for the second network packet; and based at least partly onthe modified addressing information for the first endpoint pool, routingthe second network packet to an individual endpoint computing devicewithin the first endpoint pool.
 7. The method of claim 6, whereinperiodically retrieving the configuration information comprisesperiodically retrieving the configuration from network-accessiblestorage location remote from each access point.
 8. The method of claim6, wherein the method is implemented based on execution ofcomputer-executable instructions on each access point stored withinvolatile memory, without reference to information in any non-volatilememory of each access point.
 9. The method of claim 6, wherein theplurality of access points provide access to a plurality ofnetwork-accessible services associated with network addresses within atleast first and second network address pools, and wherein the methodfurther comprises, at each access point: advertising, to a neighboringdevice of a first neighboring network of the access point, the firstnetwork address pool as reachable via the access point, withoutadvertising to the neighboring device of the first neighboring networkthat the second network address pool is reachable via the access point;and advertising, to a neighboring device of a second neighboring networkof the access point, the second network address pool as reachable viathe access point, without advertising to the neighboring device of thesecond neighboring network that the first network address pool isreachable via the access point.
 10. The method of claim 6, whereinrouting the first network packet to the individual endpoint computingdevice within the first endpoint pool comprises applying network addresstranslation (NAT) to the first network packet.
 11. The method of claim10, wherein applying network address translation (NAT) to the firstnetwork packet does not modify a transmission control protocol (TCP)sequence number within the first network packet.
 12. The method of claim6, wherein the modified configuration information further reflects anassociation of the network-accessible service with an additional networkaddress, and wherein the method further comprises: receiving a thirdnetwork packet addressed to the additional network address; selectingthe first endpoint pool, from the plurality of endpoint pools, as adestination for the third network packet; and based at least partly onthe modified addressing information for the first endpoint pool, routingthe third network packet to an individual endpoint computing devicewithin the first endpoint pool.
 13. Non-transitory computer-readablemedia include computer-executable instructions that, when executed oneach access point of a plurality of access points to anetwork-accessible service associated with a network address, configureseach access point to: periodically retrieve configuration informationshared among the plurality of access points, the configurationinformation including at least addressing information for each endpointpool of a plurality of endpoint pools, the addressing information eachendpoint pool including addresses of individual endpoint computingdevices within the endpoint pool configured to provide thenetwork-accessible service, wherein the configuration informationincludes information regarding multiple subsets of network-addressesaccessible via each access point, and wherein the network address isincluded within a first subset of the multiple subsets; implement aplurality of isolated flow managers, different flow managers configuredto route packets associated with different subsets of network addressesfrom the multiple subsets; at a first flow manager of the plurality ofisolated flow managers that is configured to route packets associatedwith the first subset; receive a first network packet addressed to thenetwork address; select a first endpoint pool, from the plurality ofendpoint pools, to which to route the first network packet; and based atleast partly on the addressing information for the first endpoint pool,route the first network packet to an individual endpoint computingdevice within the first endpoint pool; receive modified configurationinformation for the access point, the modified configuration informationreflecting modified addressing information for the first endpoint poolas a result of a change in the one or more endpoint computing devicesincluded within the first endpoint pool; and at the first flow manager:receive a second network packet addressed to the network address; selectthe first endpoint pool, from the plurality of endpoint pools, as adestination for the second network packet; and based at least partly onthe modified addressing information for the first endpoint pool, routethe second network packet to an individual endpoint computing devicewithin the first endpoint pool.
 14. The non-transitory computer-readablemedia of claim 13, wherein the computer-executable instructions areexecutable on each access point without requiring the access point toinclude non-volatile memory.
 15. The non-transitory computer-readablemedia of claim 14, wherein the computer-executable instructions arefirst computer-executable instructions, and wherein the media furthercomprises second computer-executable instructions that, when executed byeach access point, configure each access point to: retrieve the firstcomputer-executable instructions from a network-accessible storagelocation remote from the access point; place the firstcomputer-executable instructions into a volatile memory of the accesspoint; and execute the first computer-executable instructions from thevolatile memory.
 16. The non-transitory computer-readable media of claim14, wherein the plurality of access points provide access to a pluralityof network-accessible services associated with network addresses withinat least first and second network address pools, and wherein thecomputer-executable instructions are further executable by each accesspoint to: advertise, to a neighboring device of a first neighboringnetwork of the access point, the first network address pool as reachablevia the access point, without advertising to the neighboring device ofthe first neighboring network that the second network address pool isreachable via the access point; and advertise, to a neighboring deviceof a second neighboring network of the access point, the second networkaddress pool as reachable via the access point, without advertising tothe neighboring device of the second neighboring network that the firstnetwork address pool is reachable via the access point.
 17. Thenon-transitory computer-readable media of claim 14, wherein to route thefirst network packet to the individual endpoint computing device withinthe first endpoint pool, the computer-executable instructions furtherconfigure each access point to apply network address translation (NAT)to the first network packet.
 18. The non-transitory computer-readablemedia of claim 17, wherein applying network address translation (NAT) tothe first network packet does not modify a transmission control protocol(TCP) sequence number within the first network packet.
 19. Thenon-transitory computer-readable media of claim 14, wherein the modifiedconfiguration information further reflects an association of thenetwork-accessible service with an additional network address, andwherein the computer-executable instructions further configure eachaccess point to: receive a third network packet addressed to theadditional network address; select the first endpoint pool, from theplurality of endpoint pools, as a destination for the third networkpacket; and based at least partly on the modified addressing informationfor the first endpoint pool, route the third network packet to anindividual endpoint computing device within the first endpoint pool.